Web   ·   Wiki   ·   Activities   ·   Blog   ·   Lists   ·   Chat   ·   Meeting   ·   Bugs   ·   Git   ·   Translate   ·   Archive   ·   People   ·   Donate

#olpc-meeting, 2008-10-30

Index | Today     Channels | Search | Join

All times shown according to UTC.

Time Nick Message
17:00 joef bjordan:windows?
17:00 mchua #LINK http://wiki.laptop.org/go/Test[…]meetings/20081030
17:00 for agenda
17:00 agenda:     * 1 Previous meeting's action items
17:00    * 2 Activity testing
17:00    * 3 Test tools
17:00    * 4 Test automation
17:00    * 5 Tests that are hard for volunteers to replicate
17:00    * 6 Learning and teaching testing
17:00    * 7 Some notes on infrastructure
17:00    * 8 Purpose/charter/mission of community test
17:00 anything else to add?
17:01 (We'll just skip #1 because we don't have any, this being the first meeting)
17:01 and we might not get through everything, but I'd like to kick off some good discussion on the things we do get to; there's always next week
17:02 joef mchua: why #5?
17:02 cjl 5 means things like collaboration?
17:03 kimquirk larger scale system testing
17:03 frandog maybe school server stuff?
17:03 kimquirk stuff most people can't do from home
17:03 mchua joef: to figure out whether how much effort the internal QA team should spend on telling the community about things that they will mostly not be able to help with (but may be interested in), and also a discussion on things like emulation, loaner programs, etc. that we can use to make it easier
17:03 right now it sounds like testing is pretty hard to get involved in unless you're one of the lucky few with an XO
17:03 we, uh, might want to fix that sometime ;)
17:03 kimquirk i think the test results and even test cases should be viewable so people can propose thoughts, etc.
17:04 (for the large scale tests)
17:04 gregdek mchua: ayup.  :)
17:04 cjb power is a fine example
17:04 mchua cjb, kimquirk: aye - let's get through activity testing real quick first, and then skip to #5?
17:04 cjb we have a giant voltmeter at 1cc that costs thousands of dollars and knows how much power each of the XO's subsystems are using
17:04 kimquirk ok
17:04 mchua cjb: noted, and stored for the next topic in a second ;)
17:04 joef kimquirk: i think this info is public already
17:05 cjb mchua: it's okay, I'm just using it as an example of why #5 is a real concern.  I'm happy to wait to get down the agenda.
17:05 mchua so, folks - activity testing for a moment, and then let's go to #5
17:05 #TOPIC Activity testing http://wiki.laptop.org/go/Test[…]#Activity_testing
17:06 one of the big things that community test can do is Activity testing, since Activity development is a community thing
17:06 (are any of the Sugar folks in here? can someone pop into #sugar and ask if they want to come in for a moment?)
17:06 joef mchua: do we have any real volunteers "on" now
17:07 mchua: ?
17:07 gregdek mchua: Done.
17:07 (Went to #sugar and advertised meeting.)
17:07 joef mchua: ...except us non-volunteers?
17:08 kimquirk joef: you should look over the list
17:08 joef kimquirk: but not everyone is "listening"... i guess
17:09 mchua it would be awesome if community test could build a framework that devs can use - maybe even acting as QA consultants, sort of, for Activity developers
17:09 to help them write up test plans
17:09 and then running community meetups to get users, even drop-in users, to test them out
17:09 gregdek hey lfaraone.  :)
17:09 mchua (and writing tools to automate them, and running them ourselves, and... well, there's much that can be done - Activity testing is a pretty big charter)
17:09 lfaraone:  you made it!
17:09 gregdek: thanks for the #sugar plug, btw
17:10 gregdek np
17:10 mchua and joef - a number of volunteers couldn't make it in today but will be reading logs and chiming in on the list later on
17:10 joef mchua: great!
17:10 mchua (it's the first meeting, so I reckoned that it would be small - we actually have a lot more people than I expected)
17:10 lfaraone mchua: Sorry about that.
17:10 kimquirk mchua: are you thinking of more semantic media wiki templates to make it easier for Activity Developers to provide info on the activity and others to provide test cases?
17:10 mchua lfaraone: no worries. We were just talking about activity testing.
17:11 kimquirk: Yes - I think the big thing is going to be lowering the activation energy it takes to get an Activity tested.
17:11 so, for instance, see http://wiki.laptop.org/index.p[…]lyze&oldid=115198
17:11 kimquirk right... but are we worried that we drive away the less experienced activity developers with so many requirements when they write an activity
17:11 mchua This is the Analyze activity, which is the first one that edsiper, frances, and I wrote up as an example for how we might be able to do Activity tests
17:12 Yeah... that's a good point - that's why I'm wondering whether the idea of community test as a "consultant pool" that new activity devs can tap ("You write the Activity, we'll help you handle the testing") might help
17:12 hey Ian_Daniher ! good to see you here
17:13 because it /is/ hard to test something that looks like this: http://wiki.laptop.org/index.p[…]lyze&oldid=115198
17:13 Ian_Daniher I'm present and interested in testing, especially testing automation, but also quite loaded with homework and technical goodies calling for my time
17:13 I have... four XOs to mail tonight too
17:13 kimquirk yeah... i think we run the risk of scaring people away if they have to be a developer just to figure out how to read, write, or execute a test case.
17:13 mchua Ian_Daniher: understood
17:13 (and totally ok)
17:13 kimquirk the test case itself is probably 'easy' to execute
17:13 Ian_Daniher s/4/3
17:14 lfaraone_ kimquirk, so we can have more than one group.
17:14 kimquirk yeah.
17:14 and the wiki page aimed at testers should be very easy to navigate and have some step by step instructions.
17:14 lfaraone_ kimquirk,  the test author group, and the "actually run the test" group.
17:14 kimquirk, Yeah.
17:14 kimquirk and the one aimed at QA - consultants can have the detailed info on how to create and work with templates
17:14 mchua would folks be interested in the process that edsiper, frances, and I went through to get the Analyze activity into "instructions! ready to test!" phase?
17:15 kimquirk yes!
17:15 lfaraone_ mchua, Yes, I would.
17:15 mchua ooh, I like the consultant-howto idea, kimquirk
17:15 cjl kimquirk: I think at some level that an absence of feedback would be likely to drive more activity authors away from manintaining
17:15 mchua anyhoo, current process: please give lots and lots of suggestions on how to improve
17:15 step 1: turn http://wiki.laptop.org/index.p[…]lyze&oldid=115198 into http://wiki.laptop.org/go/Analyze
17:15 kimquirk cjl: yep
17:16 joef kimquirk: consultant idea =1
17:16 kimquirk: +1
17:16 lfaraone_ mchua, Try http://wiki.laptop.org/index.p[…]d=115198&diff=cur
17:16 mchua (flesh out what the Activity does - this took a lot of work with edsiper, the original author, about the features and the functionality it was supposed to have, I also ended up looking through the code a fair amt to see if there were features that weren't immediately obviously exposed)
17:16 lfaraone_: ooh, even better, thanks
17:16 #LINK http://wiki.laptop.org/index.p[…]d=115198&diff=cur
17:17 step 1 had the bonus side benefit of making Analyze more usable by new people coming in and going "what does this do, how do I use it?" (we hope)
17:17 step 2 was the test case
17:17 so, turning http://wiki.laptop.org/index.p[…]lyze&oldid=166270 into http://wiki.laptop.org/go/Tests/Activity/Analyze
17:17 what we had before was a generic Sugar Activity smoke test
17:17 and what we needed was a test specific to the Analyze activity
17:18 (the generic Sugar smoke tests should be run on everything as well, since things like "Activity can start" are good - but those alone won't tell us if the Activity does what it was designed to do)
17:19 kimquirk mchua: did you actually change the test case... or just add aditional test cases?
17:19 mchua step 3, which we haven't yet done, is "advertise test case to non-test-community users and have test-community folks coordinate and run minievents to get people to test/review Activity"
17:19 kimquirk (I'd prefer the latter)
17:19 also, let's talk about the size and time required to execute any one test case.
17:20 mchua kimquirk: I split the test cases into "generic Sugar Activity Smoke Test" and "Analyze test case", but used the old Analyze test case location for the Analyze-specific test case, if that makes sense
17:20 Can move if you prefer
17:20 kimquirk My experience suggests that if a test case takes >5 minutes... it will get interrupted more often than executed; and there will be too many bugs coming from the exectuion of just one test case to monitor and track it well.
17:21 If we think about the benefit and uses of having test cases at all --
17:21 I think the biggest benefits are: to track bugs against test cases
17:21 mchua kimquirk: is there something we can put into the current (admittedly, very broadly-brush-stroked right now) "making Activities testable" process that will make sure we have appropriately bite-sized test cases?
17:21 kimquirk And to track coverage against a test plan.
17:22 mchua +1 to benefits
17:22 frandog we can always split up the large test case into bite-sized chunks to stave off interruption
17:22 mchua I'd add to that the secondary, less tangible one of "motivate Activity developers with feedback"
17:22 kimquirk I think adding some generic info to the 'starting page' for QA-consultants about the goals of the a test case and some general guidelines is probably a good idea
17:22 mchua: i think we can probably list a few primary goals and a number of secondary ones.
17:23 gregdek May I ask a question?
17:23 mchua ok, so it sounds like it would be beneficial to list this "how to get from 'there is an Activity!' to 'this Activity is community tested and gives us the benefits that kimquirk has just listed!' process" somewhere?
17:23 kimquirk when we talk about the 'quality' of a release, we'd like to be able to point to something about the coverage of testing, and the documentation (like bugs and fixes) of the release.
17:23 mchua gregdek: please!
17:24 gregdek Why did we decide to pick Analyze as our first test case?
17:25 mchua kimquirk: ooh, and if we can give volunteer developers and testers a quick way to say "my Activity quality is really high" or "I made the quality of this Activity go up by 5 points", it makes a great way to track contributions
17:25 kimquirk i like to measure things :-)
17:25 mchua to answer gregdek's question, 2 main reasons
17:25 cjl mmmmmmmetrics
17:26 lfaraone_ So we can analyze other activities :P
17:26 mchua the first is that Analyze is an Activity that can also be used by testers as a test tool, since it gives us easy access to nicely formated data we might want to use in bugreports (presence server logs, etc)
17:26 and if we test it now and tell edsiper what features we would like added, we could get More Awesome Testing Tools for hammering on later Activities
17:27 the second reason is that edsiper and frances were talking about the Analyze activity test case, and edsiper had the time to go through and help flesh out the features list and build a testcase
17:28 a lot of the Activities, right now, I think are things that would be hard to testcase without the original developer's help, or with the help of some developer who wants to spend some time looking through the code
17:28 cjb oh, we should talk (later) about Kim's ideas for a Test Activity
17:28 or maybe we did and I wasn't watching?
17:28 mchua cjb: ooh, yes - no, we haven't yet, I'll add that to agenda
17:28 cjb: we might have to do that next week though
17:28 cjb (there are two activities:  a Bug Report Activity that reports bugs with enough information to be useful, and a Test activity that dispenses a testcase and helps you to follow it and record the result somewhere useful
17:28 ok!
17:29 kimquirk (sorry I have to leave early tonight... thanks for starting these IRC meetings, mchua ... and nice job on pulling together a test case of our test cases)
17:29 mchua For instance, for the Analyze test case to make sure that the data's accurate, we wouldn't have known that we should be checking it against /proc/net/dev output without eduardo, or without somebody diving through code
17:30 joef cjb: +1
17:30 mchua so anyway, I'm talking too much :) what do you guys think about the current procedure for getting Activities tested? (it's a total strawman)
17:31 would be especially interested in cjb's thoughts as an Activity author, frances since you went through the test cases and saw how much effort it took to make them, lfaraone_ and gregdek from the "is this a good process we can get volunteers to do?" side, and Ian_Daniher from the "can this be automated?" bit
17:31 also joef, whether this matches up with any metrics that you want to use for internal qa
17:32 joef mchua: somehow...
17:32 gregdek Do we have any way of knowing which activities are the most popular?  The most frequently run?
17:32 joef gregdek: depending by what audience...
17:32 lfaraone_ gregdek, No, other than surveys
17:33 gregdek thinks.
17:33 mchua gregdek: we have the list of Activities that ships with G1G1, the lists of Activities sent to large deployments, and in some cases feedback from those deployments about which ones the kids use a lot in schools
17:33 lfaraone_ Would it be acceptible if we added something like debian's popcorn app?
17:33 mchua lfaraone_: ooh, link pls?
17:33 joef gregdek: countries and G1G1 - different
17:33 lfaraone_ #popcon
17:33 #LINK http://popcon.debian.org/
17:34 mchua (I don't know how deployments get that information about the "these activities are popular" list they send us, though)
17:34 lfaraone_ *popcon
17:34 mchua (I can ask deployment tech support, if people here are interested in finding out)
17:34 cjb mchua: I've seen some surveys where they ask the kids which activities they like/use the most
17:34 lfaraone_: popcon isn't really relevant
17:34 it records which programs are installed on which machines
17:34 mchua cjb: ooh, are those surveys posted somewhere?
17:34 cjb it doesn't record how often programs are run
17:34 lfaraone_ cjb, As well as usage data, iirc.
17:34 gregdek My sense is that people would want to test the activities that are most likely to be used.
17:35 lfaraone_ cjb, Oh.
17:35 gregdek And that they would want to use themselves.
17:35 mchua we could also use is4's stats on which Activity wikipages are most frequenly visited, but that's not really much of a correlation
17:35 lfaraone_ cjb, Well, can we have something that records usage time? (after asking the user first)
17:35 joef mchua: we will ask Eric the kid tomorrow!
17:35 cjb lfaraone_: not as far as I know, but could be wrong.  it could (and in fact, would have to) be modified.
17:35 mchua joef: that's a good first step - we need that kind of data from a lot of kids from many different places, too
17:35 cjb lfaraone_: I'm a little wary of collecting that data because I don't trust everyone who would be in power to view it to use it wisely.
17:35 lfaraone_ cjb, Oh, of course. I don't think we'd be using that package, it's just the idea of it.
17:36 cjb lfaraone_: such as a mistaken teacher who thinks a child should be punished for playing too much simcity.
17:36 bjordan (Paint activity needs love)
17:36 cjl mchua, in terms of something that can be done on wiki, some little "community testing" badges on activity pages might serve several purposes, authors might covet it as it implies "quality" (and invest a little copperation to get it, like make their friends run the tests), it also gives users some feedback on "is this gonna work".
17:36 Maybe something showing a count of the number of test results reported and linking you off to the whole testing thing to make people aware of it.
17:36 cjb our strategy for data collection in general so far has been not to collect anything that could be misused.
17:36 lfaraone_ cjb, Ok, we A) ask the student and B) encrypt the data with OLPC's public key before transmission
17:36 cjl I'm not talking UA or GoodHousekeeping, just awareness-raising of testing may be it's most impt feature.
17:36 gregdek thinks about criteria for testing love.
17:37 mchua ok, so it sounds like activity testing is a pretty big, complex problem (as expected ;) - think we can try to break it down into subproblems and make action items for each one, for next week, so we can hit agenda item #5?
17:37 cjb lfaraone_: A) the consent of a six year old doesn't really mean anything, B) a country could choose to take out the encryption part
17:37 gregdek will think more, thanks mchua.
17:37 mchua (we've got 23 mins left and I'm going to be strict about obeying the cutoff ;)
17:37 lfaraone_ cjb, or they can add it in even if we don't.
17:37 cjb sure.  but then it's them doing the work.
17:38 lfaraone_ cjb, either way, if the govt wants to spy, it can.
17:38 cjb (.. and we can choose whether to help them.)
17:38 mchua lfaraone_, gregdek, cjb  - is "how do we prioritize Activities to test?" a problem that one (or all?) of you would like to tackle for next week?
17:38 gregdek mchua: Yes pls.
17:38 lfaraone_ mchua, Yes.
17:39 mchua there are many metrics that we could use, tools to get data for those metrics, etc... it would be great to have a short proposal on the options we can have available
17:39 gregdek What mailing list do we talk on, btw?
17:39 joef mchua: +1
17:39 mchua #LINK http://lists.laptop.org/listinfo/testing
17:39 gregdek: ^^
17:39 gregdek thx
17:40 mchua ok, gregdek, lfaraone_  - one or both of you want to run that discussion at next week's meeting, have some proposal ideas for us to look at before then?
17:40 gregdek :)
17:40 I don't have any concrete proposals, just ideas to talk through.
17:40 mchua gregdek: hey, those workfome too ;)
17:40 gregdek (Mostly around identifying people who want to own a particular area.)
17:41 lfaraone_ Meh, my only idea was the evil-1984-laptop-spying-on-you thing.
17:41 gregdek (All my thoughts revolve around giving people ownership.)
17:41 mchua bjordan would be a good person to ask about how getting feedback on Activities works with deployments, since he's been out there in schools with XOs *nudges bjordan*
17:41 lfaraone_ Either that or ask schools nicely for surveys.
17:41 mchua hm
17:41 ok - I guess I'm going to ask for volunteers for 4 things for this, then
17:42 1) compile proposals/ideas/suggestions for "how to prioritize Activities to test" and lead a short discussion on that during next week's meeting
17:42 2) ditto above for "testing metrics" and how to display them in ways that would be motivators to community testers and activity developers, like cjl's idea bove
17:43 (cjl, do you want to take #2? it sounds like you've been thinking about this)
17:43 bjordan there's no formal process right now, maybe having surveys available could help bootstrap something like that?
17:43 lfaraone_ bjordan, do these need to be paper, or can I set up a google spreadsheet or something?
17:43 cjl not sure I can, got a lot I'm juggling before going away, but I'd be happy to share my ideas
17:44 mchua 3) help me sit down and build a community test wiki portal page with the current procedure that we're using to get Activities to testable-ness, so we have something concrete to look at next meeting, so it's easier to make suggestions for improvements
17:44 lfaraone_ bjordan, I mean, if we  have to survey, what format do the schools prefer.
17:44 mchua and 4) someone to work with eduardo to push out the Analyze activity testing for a first round (this is a pretty vague action item, but we can start smaller-scale - I'd like to have at least 15 people run through the test case and report results)
17:45 takers for action items?
17:45 (oh right, cjl - sorry, I forgot you were heading out to the sunny islands. no worries.)
17:45 gregdek I'll take #1.
17:45 mchua thanks, gregdek
17:46 cjl, do you think you could do a braindump on your ideas for #2 before you leave, and I'll find somebody else to lead discussion next week, and make sure you see the results?
17:46 bjordan lfaraone_: a feedback activity that fed into a google spreadsheet would be neat
17:46 cjl sure
17:46 2 and 3 are fairly related if you go with smw as the base.
17:47 mchua yeah... lfaraone_, interested in doing a SMW-smackdown with me on #3, and the tail end of #2 once cjl finishes his braindump?
17:47 frandog i've gotta go...if there's any followup needed and no takers, mchua sign me up.
17:47 cjl getting fancy about feeding the result from a smw result page back to the activity as a pretty badge is jsut icing.
17:47 lfaraone_ mchua, sure, but it'll be a while before I have the time (read: a week or so)
17:48 mchua ooh, joef and frandog just said (at 1cc) we should talk about #4 tomorrow - we have testers visiting... so I'll take point on making sure that happens and gets passed on to someone else then
17:48 lfaraone_, I'll follow up with you on #3 after the meeting - we'll figure something out
17:49 anything else on activity testing for this week?
17:49 (man, that took a while - good discussion though!)
17:49 we have 10min left
17:49 bjordan (Paint needs <3)
17:50 cjl gregdek on prioritization, I have a thought or two.
17:50 mchua bjordan: want to head up paint testing as our second activity test round then? ;) it'd be 2-3 weeks before we need anything
17:50 bjordan mchua: sure
17:50 joef mchua: i have only one 10-year-old tester visiting tomorrow...
17:50 mchua w00t
17:50 joef:  it's a start
17:50 ok, I'm going to call activity testing a closed topic for this meeting, unless there's something else somebody wants to say
17:50 we all have action items for next week on it
17:51 gregdek cjl: Love to hear it -- send me a note?  gdk at redhat dot com
17:51 mchua I'd like to get to the "tests that are hard for volunteers to replicate" topic, http://wiki.laptop.org/go/Test[…]eers_to_replicate - but I think we don't have time today
17:51 cjb: I think we might be able to keep activity testing to half an hour at next week's meeting, would you be interested in talking about that (specifically, about the Test Activity ideas that you mentioned) next week?
17:52 cjb: (or write up a braindump we can send to everyone to read and then discuss?)
17:52 cjl ok, gist is I'm not 100% confident there is a whole lot to finesse, the activities deployments (both countries and G1G1) will want to be tested, ordering within that is likely to be driven by intrinsic factors (author cooperation) and not extrinsic value or usage.
17:53 mchua (any last-minute urgent topics that we *have* to bring up during this week's meeting? otherwise I'm actually tempted to close a couple minutes early, so we have some decompression time)
17:53 cjb mchua: sure, although I think it's Kim and your idea more than mine :)
17:53 mchua cjb: I thought it was your idea and Kim's! the conversation between the two of you was the first I'd heard of it - but... ooh, we need brainstorming on it, that's what we can do next week
17:54 any last minute panic agenda items or announcements? requests? anything?
17:54 (I'll stick around here after the meeting for metafeedback on how this meeting was scheduled/announced/run/how it went, etc)
17:56 action items I have: gregdek on prioritizing activities, cjl on initial testing-metrics-for-motivation thing, lfaraone_ and myself to pair on getting portal page up with current procedure when we're simultaneously free, me and joef and frances to coordinate on getting Analyze tested
17:56 and cjb, if you wouldn't mind bouncing off some ideas on how to direct a Test Activity brainstorm, maybe with me and kimquirk, before next week's meeting?
17:57 anything else? what did I miss?
17:58 cjb sounds good
17:58 joef mchua: who will actually test Analyze?
17:58 mchua thanks cjb
17:58 joef: we'll talk about it tomorrow with your tester and frances - we'll get it tested by volunteers
17:58 ok, we're getting close to time here
17:59 joef mchua: I don't think Eric the kid is good for Analyze
17:59 mchua any objections to closing? I'll post minutes on the wiki and the mailing list, and email folks about their action items
17:59 joef: no problem - we'll figure it out tomorrow, outside the meeting
17:59 ok, meeting closing in 3
17:59 2
17:59 1
17:59 #endmeting
17:59 er
17:59 #endmeeting
11:30 mchua_ ah, *excellent.*
11:30 #TOPIC community test
11:30 #IDEA testing idea
11:31 #LINK http://meeting.laptop.org/olpc[…]0081030_1130.html
11:31 #topic community test
11:32 Huh. Logging works, but commands don't seem to.
11:32 #endmeeting

Index | Today     Channels | Search | Join

Powered by ilbot/Modified.