Web   ·   Wiki   ·   Activities   ·   Blog   ·   Lists   ·   Chat   ·   Meeting   ·   Bugs   ·   Git   ·   Translate   ·   Archive   ·   People   ·   Donate

#olpc-meeting, 2008-11-06

Index | Today     Channels | Search | Join

All times shown according to UTC.

Time Nick Message
16:59 mchua Who's here?
16:59 (While I pull up the agenda)
16:59 bjordan Brian
16:59 hi!
16:59 mchua hey bjordan !
16:59 adricnet may or may not be here. Quantum cats are tricksy.
16:59 marcopg just watching
17:01 mchua Agenda is at http://wiki.laptop.org/go/Comm[…]etings/2008-11-06
17:01    * 1 Previous meeting's action items
17:01    * 2 Activity testing
17:01          o 2.1 Community testing portal page (lfaraone, mchua)
17:01          o 2.2 Current Activity prep-for-test procedure feedback (mchua)
17:01          o 2.3 Status of current Activities being tested (mchua, frandog)
17:01          o 2.4 Prioritizing Activities to test (gregdek)
17:01          o 2.5 Displaying testing metrics in motivating ways (cjl, from notes)
17:01    * 3 Test Activity brainstorming (cjb)
17:01    * 4 Some notes on infrastructure
17:01 Any additions?
17:01 (or comments?)
17:02 cjl *
17:02 mchua cjl: invisible agenda item? :)
17:02 Note that we don't seem to have a lot of people here tonight, so we might want to defer a few of these discussions and find a time that's better for more folks to make. We'll get to that one later though.
17:03 cjl just shining a star
17:03 mchua grins
17:03 #topic last week's action items
17:03 #TOPIC last week's action items
17:04 argh, meetingbot is not an op... oh well.
17:04 gregdek to compile proposals/ideas/suggestions for "how to prioritize Activities to test" and lead a short discussion on that during next week's meeting
17:04 gregdek's not here, but we have good notes on this so we'll go over that in a sec
17:04 #  cjl to write out a first braindump on testing metrics and how to display them in ways that would be motivators to community testers and activity developers (done!)
17:04 thanks, cjl!
17:04 #  mchua, lfaraone to pair on getting a test community portal page up with the current procedure that we're using to get Activities to testable-ness, so we have something concrete to look at next meeting and it's easier to make suggestions for improvements (done! Community testing)
17:05 mchua, frandog to find someone to work with eduardo to push out the Analyze activity testing for a first round (this is a pretty vague action item, but we can start smaller-scale - I'd like to have at least 15 people run through the test case and report results) not done, we'll go over why in a sec
17:05 cjb to think about how to run a Test Activity brainstorm at next week's meeting, bouncing ideas off of people as needed (not done, I think... we can defer this)
17:05 #ACTION mchua to follow-up with outstanding action items from last week (gregdek, frances, cjb)
17:06 hey gregdek!
17:06 gregdek hullos.
17:06 Sorry I'm late.  :)
17:06 Flaky wireless at the coffeeshop.
17:06 mchua we were just wondering about your AI from last week - "compile proposals/ideas/suggestions for "how to prioritize Activities to test" and lead a short discussion on that during next week's meeting"
17:06 gregdek: no worries.
17:07 gregdek Oh, right.
17:07 Did you ever send out the email I sent to you?
17:07 mchua gregdek: ready for that part of the discussion in a few minutes? (couple things we need to get through first)
17:07 gregdek: which one?
17:07 gregdek The long email I sent to you and Kim -- you asked my permission to send it out more broadly, iirc?
17:07 mchua gregdek: oh! yes. Gotcha. You're all set then.
17:08 hey kimquirk !
17:08 kimquirk hi
17:08 gregdek key kimquirk
17:08 hey kimquirk :)
17:08 kimquirk how's it going gregdek
17:08 gregdek Well enough, thanks.
17:08 mchua gregdek: thanks for reminding me - we'll get to your email down the line tonight.
17:08 gregdek ok.
17:09 mchua Anyway, so y'all don't have to listen to me blather on, we're looking for feedback on...
17:09 #TOPIC Community testing portal page
17:09 http://wiki.laptop.org/go/Community_testing
17:09 #LINK http://wiki.laptop.org/go/Community_testing
17:09 Behold, the ugliness.
17:09 The most functional part is http://wiki.laptop.org/go/Comm[…]_testing_meetings
17:10 joef asked that community test stuff be separated from general testing things on the wiki, which is the reason for the split
17:10 comments, questions, complaints, offers of help? ;) I'll try to ask skierpage to lend a hand at making this more SMW-better, but this portal is for you folks, so tell us what you want
17:11 howdy fhop - we're taking comments on http://wiki.laptop.org/go/Comm[…]_testing_meetings and http://wiki.laptop.org/go/Community_testing
17:11 fhop I'm here...sorry I'm late!
17:12 cjl mchua, I want to see an example of an input for for a test case. :-)
17:12 s/for for/form for/
17:12 mchua cjl: point. that's actually our next agenda item, so totally want lots of comments on it. ;)
17:13 anything else that people would like to see here, or should we move on to that requested feature?
17:13 ...I'll take that as a yes ;)
17:13 work on the portal will continue, feedback always welcomed.
17:14 #TOPIC  Current Activity prep-for-test procedure feedback
17:14 My guess is that this will occupy most of our meeting tonight, since cjb isn't here for the Test Activity brainstorm (I'll move that to next week)
17:14 so I'd like to spend a good half-hour or so on this and let 'er rip
17:14 if we can
17:14 so gregdek asked me the other day how to get started doing Activity testing
17:15 my response "Er.... you.... um..." *goes off and makes http://wiki.laptop.org/go/How_[…]test_an_Activity* "...use our handy guide!"
17:15 our handy guide is currently not terribly handy
17:15 kimquirk it looks like a good start
17:16 mchua kimquirk also requested earlier that we spend some time on quality, metrics, and how people can take ownership of that for specific activities or features of Sugar, so I guess this is also the place to have that discussion
17:16 kimquirk I think we can add to the high level topics that would follow from 'decide on testing goals'
17:16 mchua gregdek: thoughts on the page? what are you getting stopped by?
17:16 kimquirk mchua: yes... that is, but we should also get the stub of the entire things
17:16 mchua cjl: does the Analyze activity give you the example of a input form for a test case that you wanted?
17:16 bjordan: can you imagine doing the same thing for, say, Paint?
17:17 cjl didn't look super close at Anayze. . .looks again.
17:17 mchua kimquirk: sorry, I'm not sure what you meant by "stub of the entire thing..."
17:17 kimquirk so, after you decide on testing goals (and test case coverage); then you create test case(s); then you ask people to help execute them
17:17 (or something like that)
17:18 mchua oh! yes.
17:18 We're definitely missing those future steps. :)
17:18 notes that these should be put in...
17:19 cjl mchua: unfortunately while the test case seems well described in scripted fashion on http://wiki.laptop.org/go/Tests/Activity/Analyze It is not at all clear to me how I would record the results of my running through that script.
17:19 kimquirk as for the testing goals... they should identify the features of the activity, and maybe we suggest stubbing out test cases that are relatively isolated and easy to implement by themselves.
17:19 And a good step by step on how to record and view results.
17:19 mchua cjl: ooh, good point. Thanks.
17:19 kimquirk (as cjl suggests)
17:20 mchua cjl: I'll fix that for Analyze... gregdek, do you think you could use that as an example once I'm done to generalize how to do it for Speak and other Activities?
17:20 cjl Test ing great and happens all the time.  recording results of testing gets you closert to something QA-ish.
17:20 mchua cjl: +1
17:21 marcopg is the idea to register results in the wiki?
17:21 mchua kimquirk: re: stubbing out test cases, do you have any preference for how we'd organize those on the wiki? We'd end up with lots of subpages, like [[Tests/Activity/Analyze/Network screen]] and [[Tests/Activity/Analyze/Presence server screen]] and...
17:21 marcopg they will need to get in trac at some point, I guess
17:21 cjl Also create opportunity for feedback on test case (testing hte test), this of course could just be handled by talk page, but then you should say "Commment on this test qua test on talk page"
17:21 kimquirk marcopg: if you 'edit with form' on a test case, then you will get an 'add another' button that lets you add results.
17:22 mchua: I think you have a good view of organzation, so what you suggest, we'll follow :-)
17:22 mchua marcopg: I think the idea is to register results in the wiki, and then if a test fails, then file bugs against that component and link to those bugs from the failed test case as a "why this test failed" indicator... kimquirk? the original form was your design
17:22 kimquirk marcopg: there is a place to add trac bug numbers in the results table
17:23 mchua: right
17:23 marcopg cool
17:23 mchua sweet. marcopg, does that sound ok for developers, or do you want more info?
17:23 kimquirk mchua: it actually wasn't my design... that's funny because I thought you were behind a lot of the original design :-)
17:23 marcopg looks pretty good to me!
17:23 mchua yay!
17:24 hm, we must track down this mysterious person behind the SMW test cases... francesca, I think...
17:24 kimquirk i think you're right
17:24 i thought she was working with you for wiki expertise
17:24 cjl marcopg, We should not think of these test cases as instantly generating a solid trac bug.  These are screening tests to be run by less skilled.  Numerous fails would draw deeper attention.
17:24 kimquirk cjl: sounds right
17:24 bjordan mchua: (re: making a testing guide for paint?) sure
17:25 gregdek ...
17:25 marcopg cjl: yup, but we should make sure they makes their way into trac, at some point
17:25 cjl yes
17:25 joef i think it was my design and Charlie, at al execution...
17:26 cjl Which is why the results should be highly visible and searchable (and watched by someone intersted in bridging those systems).
17:26 mchua bjordan: this is great - that means we have 3 Activities beta-testing the procedure in http://wiki.laptop.org/go/How_to_test_an_Activity, which means that you and gregdek and fhop are going to find a lot of my "Mel can't write" bugs there :)
17:26 joef: kudos on the design, then :)
17:26 (I like it a lot)
17:26 joef ...execution not exactly the way i wanted it... but blame that wiki!!!
17:26 mchua: thanks!
17:27 mchua actually, this might be a good time to skip over the "status of activities being tested" agenda item (because they're all in the "just getting started" phase) and talk about how to prioritize which ones to test
17:27 gregdek hrms.
17:28 mchua gregdek: thoughts? :)
17:28 gregdek The Patton quote comes to mind.
17:28 "Don't tell people how to do things.  Tell them what needs to be done, and let the surprise you with their ingenuity."
17:28 s/the/them
17:28 cjl The point is to make the other poor bastard die for his country"?
17:29 gregdek LOL.
17:29 No, the other quote.  :)
17:29 cjl ok
17:29 bjordan hehe
17:29 mchua gregdek: I thought you were going to say "A good plan violently executed now is better than a perfect plan executed next week."
17:29 joef gregdek: exactly how our system of test cases was created!
17:29 gregdek I'm a bit concerned about building test cases with the expectation that novices are going to spend 10 minutes testing and end up doing something useful.
17:30 adricnet For varying definitions of novice..
17:30 Without test cases, can any useful data be collected or harvested ?
17:31 fhop gregdek: what concerns you?
17:31 gregdek Well...
17:31 mchua I was going to ask for ideas on how to recruit large numbers of unskilled short-term testers, but... gregdek raises a good point. to put it bluntly, community testers are bloody brilliant people; should we ask them to spend their time recruiting drones?
17:31 being a drone isn't much fun. Managing drones... not really fun either.
17:31 gregdek ...it seems to me that the work of building the test cases, and the work of executing test cases, are (a) going to widely vary by activity, and (b) for a given activity, will represent a 90% overlap between "planning" and "executing".
17:31 joef adricnet: yes.
17:31 gregdek It seems to me...
17:31 fhop gregdek: ...that they won't be able to communicate what they find or that it won't be worthwhile in the long run?
17:32 gregdek ...that one person who is *dedicated* to testing *one activity* will be able to do almost everything that is required to keep that activity well-tested.
17:32 cjl There is some weight to the argument to that if you can describe a test precisely enough for a total n00b, you can probably jsut automate it instead.
17:32 joef adricnet: ...it's called "ad hoc testing"
17:33 fhop gregdek: i see...better results from more intense "fiddlings"
17:33 mchua gregdek: ...and that OLPC should bribe them with an XO if they maintain an Activity that we ship for the length of a release cycle? ;)
17:33 gregdek I don't want shallow testing.  I want deep testing.
17:33 mchua: Maybe.  :)
17:33 mchua cjl: agreed. we have, uh... some work to do on making automation possible for Activity testing.
17:33 bjordan an example of a bug that a strict functionality test case might not catch (but deep testing might): http://dev.laptop.org/ticket/8864
17:33 fhop gregdek: however, with that said, some of the shallow testing can point us to where the more in depth testing should be directed
17:33 mchua marcopg: which reminds me that I probably should start hitting the #sugar channel and asking about sugarbot and etc.
17:34 adricnet joef: It is, and I do it :) .. want to develop better sol'ns and automate all the boring stuff
17:34 gregdek fhop: We can give very high-level general guidelines for all activities.
17:34 joef adricnet: automation isn't cheap
17:34 marcopg mchua: sugarbot is on my list of things to do but I'm not sure whenever I'll get at it :(
17:34 mchua howdy mvn071, and welcome - in the middle of the test community meeting right now, http://wiki.laptop.org/go/Comm[…]etings/2008-11-06
17:34 marcopg mchua: so help totally appreciated :)
17:34 gregdek Like: who's the owner? how do I file a bug?  How do I record results?
17:34 mchua marcopg: sugarbot is now on my list of things to do also, so I should be getting around to it :)
17:34 gregdek But building a full test case is *hard*.
17:34 adricnet joef: No, but it's easier to keep bots interested and fed than volunteers :D
17:35 marcopg mchua: yay :)
17:35 adricnet: you also need to keep developers interested in writing testcases
17:35 gregdek I'm not sure I agree with that.
17:36 I'm not sure that developers will ever be properly incented to write test cases, ever.
17:36 mchua adricnet: are you interested in working on an automation system? I could totally use a hand, especially in coming up with a good design (as in, "I'm a tester. I want to automate this boring thing. What is my ideal interface to do so / the most beautiful tool I could imagine for it?")
17:36 adricnet marcopg: A worthy goal as well
17:36 mchua: I want to help you dream it up, yeah. We may have trick someone else into coding it :D
17:36 marcopg gregdek: it's tricky for sure
17:36 mchua adricnet: *grin* I've found that if you spec things out well enough, finding a volunteer to code it is the easy part. ;)
17:36 marcopg I was pointing out exactly that problem there
17:37 mchua adricnet: so dreaming help is what's needed most atm.
17:37 Since we're still on the topic of "how to test Activities," I wanted to bring up something that we've briefly discussed in internal QA before - exploratory testing.
17:38 #LINK http://en.wikipedia.org/wiki/Exploratory_testing
17:38 this is the least drone-like way of testing that I know of
17:38 marcopg I like the idea of starting community testing
17:38 s/community/automated
17:38 mchua it takes as its thesis exactly what gregdek put forth about planning and executing tests having a huge amount of overlap
17:39 marcopg but I'm convinced in the short time we can get much better results improving community testing
17:39 bjordan mchua: I've got some spec'd out bugs for Paint I could use some help finding a volunteer to code/commit in that case :)
17:39 marcopg developers time is too much of a precious resource atm
17:39 mchua and what I like most about it is that it starts with the assumption that testers are Really, Really Smart and that to use anything less than their full talents on whaling on a piece of software is a crime
17:39 gregdek My take is simple.
17:39 mchua bjordan: cool-o, point me towards them in email?
17:39 bjordan mchua: fo sho
17:39 mchua bjordan: thanks!
17:41 waits to hear gregdek's thoughts
17:41 gregdek Oh, I got distracted, sorry.  :)
17:41 marcopg hehhe /me waiting too
17:42 gregdek LOL!
17:42 OK.
17:42 Simple thought #1:
17:42 It all starts with a passionate person.
17:42 Simple thought #2:
17:42 Better to get excellent testing for one or two activities than crappy testing for a ton of activities.
17:42 Simple thought #3:
17:43 The people who are going to have the best input are going to be the testers, NOT the developers.  :)
17:43 Which means, to me:
17:43 Simple action plan.
17:43 #1. Advertise for testers.
17:43 #2. Put them in charge of figuring all this out for one activity each.  If you find 3 folks, that's probably great.
17:43 #3. Write the tools that *they* tell you to write.
17:44 Oh, and #4. make sure that they have incredibly close relationships with their activity developers from day one.
17:44 That's it.
17:44 mchua likes #3 and #4 especially
17:44 joef gregdek: i'm on your side!
17:44 gregdek trac stuff, wiki plans, all the other stuff: let the testers drive that stuff.
17:44 marcopg heh gregdek is wise as usual
17:45 cjl mchua gregdek that begins to narrow down the group you can recruit from (very small community), and most of them are probably trying to devel something on their own anyway and only want to test things that get in their way (i.e. not activities).
17:45 gregdek cjl: Not sure I agree there either.
17:46 There are plenty of smart people who want to help.
17:46 cjl I'm not saying that gregdek is worong, these highly engaged testers will give you the best results, the querstion is can you do thta an do something broader.
17:46 gregdek They just can't find anything they can really drive.
17:46 I'm a firm believer in focus, and developing the right model.
17:46 marcopg cjl: in my experience even just one very good tester can make a huge difference
17:46 gregdek And starting small, and being HIGHLY effective at one thing.
17:47 cjl gregdek: I like the way it sounds, question is why hasn't it happened yet?
17:47 and what happens when tha one good individual moves along.
17:47 mchua is thinking of the test group of high school students that Caroline Meeks is trying to build locally in Boston... if we can get each student - or a student pair - to take on an Activity, and give them access to good mentors...
17:47 marcopg cjl: it started to happen
17:47 cjl: at the end of 8.2 cycle we had some really excellent testing, by community people
17:48 gregdek cjl: Because we haven't asked in the right way.  :)
17:48 Give people ownership of something -- say "this problem domain is yours, we're counting on you" -- and *amazing* things happen.  Amazing things.
17:48 cjl I'm not trying to fight with you guys, just challenge to test the strength of the ideas  (cjl = idea tester)
17:48 joef gregdek: right
17:49 gregdek And activity QA is enough out of the critical path, and not getting done anyway, that it's safe to do so.
17:49 Why do people get involved with OLPC?  Folks who aren't getting paid, anyway?
17:49 adricnet gregdek: Masochism?
17:50 the Mission ?
17:50 gregdek adricnet: Really?  :)
17:50 mchua marcopg: what happened? or is it continuing to happen?
17:50 votes masochism!
17:50 joef gregdek: ownership of "something"
17:50 gregdek Because they want to help, and they want to be recognized.
17:50 They want to help you guys change the freaking world.  :)
17:50 mchua notes that there are 9 mins left in meeting time tonight and that we haven't hit on two topics
17:51 # 2.4 Prioritizing Activities to test (gregdek)
17:51 # 2.5 Displaying testing metrics in motivating ways (cjl, from notes)
17:51 gregdek Oh, ok.  :)
17:51 marcopg mchua: after release as usual we had a big break on people involvement
17:51 joef gregdek: by owning a piece of the "process"
17:51 mchua (gregdek, cjl, ok to push those discussons to email? that way we can keep talking about this)
17:51 gregdek I've had my say anyway.
17:51 Find passionate people and give them ownership of problems.  :)
17:51 cjl sure
17:51 marcopg mchua: which I think it's a problem we need to address
17:52 mchua marcopg, what if we used one of gregdek's ideas and told people
17:52 "Ok - if you spend these 6 months pre-release testing an Activity, and you get it tested well enough that we ship it"
17:52 "We'll list you as The Tester!!! in the thank-you notes for that build, *and* we'll ship you an XO of your own for free"
17:53 gregdek They'll need the XO in the first place to test.  :)
17:53 mchua "if you agree to, for the next 6 months (of the release cycle) maintain the testing and act as the community focal point for keeping that Activity good"
17:53 fhop ...and I'll shake some hands, kiss some babies
17:53 bjordan mchua: re: 2.4, I suggest we start with G1G1 activities, maybe poll which ones people think are most riddled with solv-able bugs?
17:53 fhop SPEAK!
17:53 gregdek I'VE GOT SPEAK.
17:53 bjordan speak has little to no bugs
17:53 gregdek I HAVE CLAIMED SPEAK.
17:53 bjordan :-p
17:53 gregdek There is a bug, you know.
17:53 bjordan compared to the other g1g1 activiteis
17:53 fhop its just a major bummer when it doesn't work
17:53 gregdek "Swahili" is listed as "Swahihi" in the language list.  ;)
17:53 mchua laughs
17:54 bjordan yeah, that is a really funny one
17:54 marcopg mchua: it might work... I'm wondering though, why it doesn't work much for activity maintainers itself
17:54 bjordan is that in trac, actually?
17:54 mchua looks at clock
17:54 gregdek bjordan: I was waiting until I had a process document to follow.
17:54 Sorry mchua.  :)
17:54 marcopg mchua: they have ownership of the activity, but most of them are not doing a great a work at it lately
17:54 gregdek (And also, is there a speak component in trac?)
17:54 adricnet I can add some RFE bugs to Speak in a hurry ;)
17:54 mchua gregdek: thanks for making me start in on http://wiki.laptop.org/go/How_to_test_an_Activity
17:55 looks at the time
17:55 gregdek marcopg: Maybe if there were a dedicated QA tester on their ass, it would give them more incentive.  ;)
17:55 marcopg I'm trying to understand the reason
17:55 fhop i can be really mean
17:55 marcopg gregdek: that's actually very true
17:55 mchua gregdek: the question becomes how we motivate the QA people, then ;)
17:55 fhop i'm a teacher ;)
17:55 gregdek marcopg: I think that a lot of Activity testers feel like no one is paying attention to what they do.
17:55 s/testers/writers/
17:55 Sorry
17:55 marcopg yup
17:55 gregdek marcopg: I think that a lot of Activity *writers* feel like no one is paying attention to what they do.
17:55 So finding truly dedicated testers could create a virtuous circle.
17:56 adricnet might well be true. Lots of activities , and only some are popular
17:56 gregdek Right.
17:56 No one wants to feel like no one cares about what they do.
17:56 mchua folks, I hate to say this, but we need to wrap up (time's a-tickin')
17:56 how can we channel this convo into email?
17:56 do you want to continue the conversation and then summarize to list?
17:56 cjl CTRL-C, CTRL-V
17:56 mchua I think a lot of *great* points have been brought up in the last *backscrolls* 10 minutes especially
17:57 gregdek So lemme address my next point -- prioritization?
17:57 mchua gregdek: email? time! ;)
17:57 gregdek: or next week's Big Discussion, if you like
17:57 gregdek OK.
17:57 :)
17:57 mchua gregdek: thanks
17:58 so here's what I think we can do
17:58 bjordan mchua: can you send out a Call for Emails email? :)
17:58 mchua gregdek and bjordan can try out http://wiki.laptop.org/go/How_to_test_an_Activity out on Speak and Paint respectively, and complain loudly when things are missing, mchua to revise
17:58 bjordan: I will
17:58 bjordan: probably in a few hours though
17:58 * adricnet and mchua will dream up an interface for automated testing, trying to eliminate our need for drones
17:58 adricnet I'd enjoy jawing with someone about automated testing of at least browse using Mozilla tools.
17:58 mchua adricnet: feel like tackling Browse then? ;)
17:59 adricnet is always working itself out o a job. that's part of the point of working.
17:59 mchua I will talk with sj, kimquirk, and others about what kinds of brib^H^H^H^H resources we can offer testers, for motivation
17:59 adricnet mchua: Well there's enough room there for plenty of us. Dragons too.
17:59 fhop mchua cookies
17:59 bjordan hahah
17:59 fhop and whippings
18:00 mchua we'll look at the process / motivation-tools at next week's meeting, refine, and then try to recruit those Focusedly-ish Dedicated Peoples
18:00 bjordan whipped creme cookies
18:00 fhop (thats not so much incentive, but keeps 'em in line)
18:00 mchua does that sound ok?
18:00 joef beer and wine!
18:00 fhop joef vodka!
18:00 joef or ... Russian vodka!
18:00 cjl gets Caribbean vacations for testing. . .
18:00 mchua I'm going to take the excited comments about testing incentives as an "yes, that sounds ok" ;)
18:01 fhop wishes cjl would take me in his suitcase
18:01 mchua ok - I'll send out... lots of emails tonight about this, then, to set things up
18:01 is excited!
18:01 meeting ending in a few... any last thoughts?
18:01 3
18:01 2
18:01 1
18:01 #endmeeting

Index | Today     Channels | Search | Join

Powered by ilbot/Modified.
Webmaster