Time |
Nick |
Message |
17:02 |
adricnet |
pong. |
17:02 |
mchua |
Wow. Thanks, guys. I was in the wrong room... |
17:02 |
skierpage |
can operator /topic http://wiki.laptop.org/go/Comm[…]etings/2008-11-20 ? |
17:03 |
mchua |
hi garycmartin, adricnet, skierpage, fhop! |
17:03 |
gregdek |
...aaaaand /me has to bug out because we're being kicked out of this physical room. |
17:03 |
garycmartin |
bloop.... |
17:03 |
gregdek |
Dammit. Sorry folks. |
17:03 |
mchua |
no, my mistake. |
17:03 |
|
Anyhow, http://wiki.laptop.org/go/Comm[…]etings/2008-11-20 |
17:03 |
garycmartin |
room? |
17:03 |
mchua |
garycmartin: ja. |
17:03 |
|
* 1 Previous meeting's action items |
17:03 |
|
* 2 Time change |
17:03 |
|
* 3 Requests for testing |
17:03 |
|
* 4 Test Jam |
17:03 |
|
* 5 Friends in testing vs. Community testing |
17:03 |
|
any additions? |
17:04 |
|
(I'm going to see if we can wrap this up in 30min because I know a number of people are pressed for time today. I'll be around afterwards for people who want to discuss things.) |
17:04 |
|
last call for additions |
17:04 |
skierpage |
mchua, did you want to talk about http://wiki.laptop.org/go/Comm[…]n_motivating_ways |
17:05 |
bjordan |
#LINK http://wiki.laptop.org/go/Tests/Activity/Paint |
17:05 |
mchua |
skierpage: Yes! We'll get to that in a few minutes; |
17:05 |
|
skierpage: I think it will be our longest discussion so I'd like to have it last |
17:05 |
|
bjordan: thanks! |
17:06 |
|
* 6 testing metrics (skierpage) |
17:06 |
|
other additions? |
17:06 |
|
I'll launch into #1 by showing a link: http://wiki.laptop.org/go/Comm[…]1-13#Action_items |
17:06 |
|
a lot of things are still ongoing, so unless someone is really itching to take on more work this week, we have enough to keep busy, I think ;) |
17:07 |
cjl |
use link command if you want the bot to hear it :-) |
17:07 |
mchua |
#LINK http://wiki.laptop.org/go/Comm[…]1-13#Action_items |
17:07 |
|
thanks cjl |
17:07 |
|
of note: the wellington test group == awesome (has everyone seen their notes? I'll link them in the weekly summary) |
17:08 |
|
also of note: anna_bham has done some cool XS hacks but everything is privately documented right now - we need to find a way for people with private notes to gently migrate them to public |
17:08 |
garycmartin |
yea,fab stuff.. |
17:08 |
mchua |
examples of good "blog to wiki" migrations welcome/wanted (it is an internal blog) |
17:08 |
|
oh! garycmartin, you reminded me |
17:08 |
|
* 7 sugarbot update |
17:09 |
|
the other thing on the action items list was activity testing - bjordan, I see you've been doing some stuff on Paint? |
17:10 |
|
gregdek and Caryl are incredibly busy with G1G1 stuff this week (fedora, support-gang, knowing them probably a million other things while simultaneously running communities to save the world), so I believe bjordan is our Activity test dude of the week |
17:10 |
|
bjordan: any thoughts? |
17:10 |
|
bjordan: ping |
17:10 |
|
(if bjordan comes back in later, we'll pop back to this for a bit) |
17:10 |
|
anything else on action items? |
17:11 |
|
okeydokey, #2 - time change |
17:11 |
|
the wellington testers have requested moving this 1hr later, to 2300 UTC / 6pm EST / etc. so that they can attend during their lunch break |
17:11 |
|
(it is 11:12 AM for them right now) |
17:11 |
|
(on Friday) |
17:11 |
garycmartin |
f |
17:11 |
mchua |
any objections? I'll send this to the list to make sure we don't have absentee collisions, but does this pose a problem to anyone? |
17:12 |
bjordan |
mchua: yup, got something of a test description up |
17:12 |
cjl |
+1 (from -5UTC) |
17:12 |
garycmartin |
mchua: fine for me... |
17:12 |
mchua |
ah, bjordan! can you give us a quick update on paint? |
17:12 |
|
(popping back to action items, agenda item #1 for just a moment...) |
17:12 |
bjordan |
mchua: and made the Paint wiki page reflect reality (instead of old UI mockups) |
17:12 |
mchua |
(but feel free to +1 / complain about the 1-hr-later meeting time if needed, since that's quick) |
17:12 |
|
bjordan: awesome, thanks! |
17:13 |
|
bjordan: were you using http://wiki.laptop.org/go/How_to_test_an_Activity ? |
17:13 |
bjordan |
mchua: nope :-X |
17:13 |
mchua |
bjordan: no worries :) can you take a look at that for next round of testing? |
17:13 |
|
rkabir: welcome, ryan! |
17:13 |
bjordan |
for sure |
17:14 |
mchua |
rkabir: this is the community test meeting, good to have you here :) |
17:14 |
|
bjordan: thanks! |
17:14 |
|
#link http://wiki.laptop.org/go/How_to_test_an_Activity |
17:14 |
|
#LINK http://wiki.laptop.org/go/How_to_test_an_Activity |
17:14 |
|
I think that takes care of agenda items 1-2 |
17:14 |
bjordan |
mchua: it has some good stuff, close to what I was doing (but I'll change to the structure described there) |
17:14 |
mchua |
bjordan: oh! sorry, didn't meant to cut you off |
17:14 |
skierpage |
bjordan, did you use http://wiki.laptop.org/go/Reporting_test_results to update the test case with your testing? |
17:14 |
rkabir |
mchua: thanks, thought i'd lurk, don't mind me! |
17:15 |
bjordan |
skierpage: yup, used "edit with form" on the Paint test case page |
17:15 |
cjl |
mchuia, time change #2 isn't done until you issue #agreed blah blah |
17:16 |
mchua |
cjl: noted. will do in just a min when bjordan is done talking about activity testing :) |
17:16 |
|
...or _bjordan , who is apparently his doppelganger |
17:16 |
_bjordan |
sorry, bjordan's irc is lagging and bouncing out from England |
17:16 |
mchua |
_bjordan: bjordan: np. :) anything else on activity testin'? |
17:16 |
|
Paint's page looks a sight better than it was last week |
17:17 |
|
wonders if bjordan's IRC is still laggy ;) |
17:17 |
_bjordan |
I found it useful to perform a thorough full functionality test and view all open bugs for the activity before/while editing the wiki page and test case |
17:18 |
mchua |
_bjordan: noted, and will add to the howto page |
17:18 |
skierpage |
bjordan, mchua, I notice that the embedded Test result doesn't capture testers name, time testing, or activity version. |
17:18 |
|
those were things you mentioned wanting on that metrics page. For now I guess people can use comments. |
17:18 |
mchua |
skierpage: yep, we should definitely change that |
17:18 |
|
skierpage: thanks for the catch |
17:19 |
_bjordan |
skierpage: yeah, that would be useful |
17:19 |
mchua |
actually, anyone mind if we zoom through our other 2 agenda items + get a quick garycmartin update on sugarbot so skierpage can talk about the testing metrics work of awesome that's been done this week? |
17:19 |
_bjordan |
sure |
17:19 |
cjl |
go |
17:19 |
mchua |
agenda #3 is very quick. |
17:19 |
|
#LINK http://invisible.net/openlibra[…]c/bookreader.html |
17:19 |
|
they have requested testers for it |
17:19 |
|
there are no test cases right now, and the use case is vague |
17:20 |
skierpage |
I wonder if it would be useful to have a page about generic activity testing --keep, save, see in journal, etc. |
17:20 |
mchua |
this doesn't seem like super high priority yet, but if someone has a lot of free time or is interested, it's out there |
17:20 |
|
that's all for #3 |
17:20 |
|
#4 is test jam - Caryl isn't here, so that's deferred to next meeting |
17:20 |
|
#5 is merging friends in testing & community testing |
17:20 |
skierpage |
mchua, is there an activity page for #3 to associate test case/test results |
17:20 |
_bjordan |
OT remote testing usefulness: |
17:20 |
|
#LINK http://dev.laptop.org/~bjordan[…]stallingx11vnc.sh |
17:21 |
mchua |
#5 is skierpage 's suggestion, and I personally think it is a good one |
17:21 |
|
any objections to it? |
17:21 |
|
skierpage: there isn't, which is another reason why it's low priority; they said "test this" but didn't give us anything other than that |
17:21 |
|
skierpage: I told them that they needed to explain what needed testing and give us something to work with for test cases, or somebody to work with to make them, so far have not heard back |
17:22 |
|
in any case, objections to merging [[Friends in testing]] and [[Community testing]]? |
17:22 |
garycmartin |
sugarbot, sorry no new testing results here (work). But will try for next meet. |
17:22 |
mchua |
it will take a little bit of talking with m_stone and joef to make sure they're ok with the merge, since [[Friends in Testing]] was originally made for the 8.2 release and the two of them drove that page |
17:23 |
|
but I think it should be ok |
17:23 |
|
garycmartin: np :) there's been a lot going on here too |
17:23 |
|
any volunteers for the [[Friends]] / [[Community]] merge? |
17:23 |
|
I'll shout out on the mailing list to see, too |
17:23 |
|
tabitha_: hello! |
17:24 |
|
tabitha_: meeting.laptop.org has notes so far |
17:24 |
tabitha_ |
hello |
17:24 |
skierpage |
mchua, I'll do it. But are you ready to have people hit the wiki, go to Friends in testing, and start testing activities? |
17:24 |
mchua |
tabitha_: we're actually looking like we will be ending early tonight, because the meeting is small / a lot of people are busy this week, with holidays |
17:24 |
tabitha_ |
right |
17:24 |
mchua |
skierpage: Not in floods, yet. |
17:25 |
|
skierpage: We don't yet have tools/procedures/infrastructure in place to take in lots of newbie testers, which is why we've been ramping up gently with http://wiki.laptop.org/go/How to test an Activity and such. |
17:25 |
tabitha_ |
well quick update from me - Alastair and I had quick chat earlier this week, we will devote some small resource to book reader testing, but still main focus on G1G1 activity testing tomorrow |
17:26 |
mchua |
skierpage: so maybe we should put a sign up saying "you're welcome to join - a lot of things are under construction and might be chaotic, but we're happy to welcome and mentor newcomers if you want to dive in" |
17:26 |
skierpage |
mchua, OK, I'll make Friends in testing say "ramping up gently". There's definitely two testing areas/interests -- "I want to run bleeding edge" and "I'll let people know what I find running activities". But the first is dormant right now |
17:26 |
mchua |
tabitha_: sounds great, thanks for the update! would love to hear more about what you're doing after skierpage talks about metrics |
17:27 |
|
skierpage: Yep. |
17:27 |
|
skierpage: Thanks tremendously - you're doing a lot this week! |
17:27 |
|
aly: welcome! |
17:27 |
aly |
hi all |
17:27 |
mchua |
aly: tabitha_ just gave us a short update of what you've been up to, skierpage is going to talk about metrics in a sec and then I'd love to hear more from the two of you |
17:27 |
fhop |
hi NZers! |
17:27 |
adricnet |
reads scrollback so far around a design meeting, oops phonecall, ... yeah |
17:27 |
mchua |
looks at agenda |
17:28 |
bjordan |
runs to harvard for nicholas' talk&such |
17:28 |
tabitha_ |
I like the idea of mentoring new testers |
17:28 |
bjordan |
bey all |
17:28 |
mchua |
I think we pretty much have everything except for metrics and... |
17:28 |
|
...and the time |
17:28 |
|
tabitha_, aly have asked to move this an hour later, so far no objections |
17:28 |
|
any last minute objections? |
17:28 |
|
bjordan: bey yourself ;) |
17:29 |
|
I think we're pretty good on that front (everyone else here has agreed, more or less) |
17:29 |
|
tabitha_, aly: meeting time has been moved for you :) |
17:29 |
|
#AGREED meetings now start at 2300 |
17:29 |
tabitha_ |
Thanks Mel, that will make it much easier for us to join from here in NZ |
17:29 |
mchua |
so - thanks for waiting, skierpage - *now* we can talk about metrics! |
17:29 |
|
#LINK http://wiki.laptop.org/go/Comm[…]ys#Implementation |
17:30 |
|
go for it, skierpage |
17:30 |
skierpage |
So I think test cases in the wiki is working pretty well. |
17:30 |
|
But storing test results in a test case has problems. |
17:31 |
|
You can't get good metrics from a test case because all you're saying is Tests/Activity/Paint passed and failed and passed and this date and that date |
17:31 |
mchua |
Storing test results, or our current implementation of storing test results? |
17:31 |
skierpage |
You have to go to the test case to see that information in rows. |
17:31 |
|
mchua, yes the current implementation |
17:32 |
|
So one could add the missing activity version/time spent/user name fields to [Add another] test result, but you'll never get metrics from it. |
17:32 |
|
So maybe doing test results somewhere else is better... like a spreadsheet. So look at |
17:33 |
|
http://spreadsheets.google.com[…]R10gwSAHxMTv6VfEA , or the view only in the link mchua gave out. |
17:34 |
|
I'm happy for others to make the choice between these. I'm willing to keep hacking on the spreadsheet and try making Google's form interface to adding a spreadsheet row work nicely. |
17:35 |
tabitha_ |
Think we need more detail but could be link to separate worksheet with overall page as you have done already |
17:36 |
skierpage |
What I'm not good at is spreadsheet jockey hacking. Someone at OLPC a few months ago said they were... I think it was in a conversation with greg smith? |
17:36 |
mchua |
I could believe that. |
17:36 |
aly |
i think it would be good to have all testing with regard to a certain activity separate |
17:36 |
mchua |
aly + 1 |
17:36 |
tabitha_ |
It is hard to remember all the things to check for, have to remember to check all functions on its own, wiki stuff exists, how it works collaborating - want to go "tick" next to each thing |
17:37 |
aly |
so then you can look back and see the history |
17:37 |
mchua |
skierpage: I think you point out a good issue - that we don't necessarily have a good idea of the information that we /want/ to capture for each Activity |
17:37 |
|
which makes it harder to talk about where to capture it (although the spreadsheet is certainly way more readable/manipulable than the current form output) |
17:37 |
skierpage |
aly, maybe. I think it would be hard to aggregate testing metrics across multiple spreadsheets. Maybe each activity could be a separate sheet in one spreadsheet). |
17:38 |
aly |
yep that sounds like it could work |
17:38 |
skierpage |
As you can see I was able to transfer test results from current test cases, but it was painful to guess the date from wiki history. |
17:39 |
mchua |
Yeah... thanks for doing that legwork, skierpage. |
17:39 |
skierpage |
So I think y'all want to make the choice (current test results in test case vs. test results in Google docs) before you invite large user testing. |
17:39 |
|
Hmmm. |
17:39 |
cr_lf |
Carl K here... one of the questions I have is do we have functional requirements for any of the activities?... reason for asking is that a lot of the test metrics should come from those... |
17:40 |
mchua |
what is the best way to get a tentative metric / results entry form settled fast, so that Activity testers can charge onwards? |
17:40 |
|
cr_lf: kind of |
17:40 |
|
(and welcome, Carl!) |
17:40 |
|
cr_lf: One of the big blockers to getting Activities tested is that we *don't* have functional requirements / feature lists / any documentation for a lot of Activities |
17:40 |
cr_lf |
one concern leading from that is that if we are testing for screen rotation and the progarm was never designed to do that then it will always fail... |
17:40 |
mchua |
cr_lf: that's why http://wiki.laptop.org/go/How_to_test_an_Activity has a big "make an oracle" section at the beginning |
17:40 |
|
yeah. |
17:41 |
cr_lf |
ah... so we write our own functional requirements and the devs then have to implement? |
17:41 |
skierpage |
mchua, if there were definite metric demands from the PHBs then it would probably eliminate the current way of doing test results. |
17:41 |
cr_lf |
I like it!!! |
17:41 |
skierpage |
are there any PHBs at OLPC :-) ? |
17:41 |
mchua |
cr_lf: ideally we'd work with the developers to figure out what the functional requirements they intended to meet were, but that would sure be nice :) |
17:41 |
|
skierpage: we have no such definite metric demands |
17:41 |
aly |
would it make sense to have a way to disable the screen rotation for apps that dont support it ? |
17:42 |
mchua |
lack of PHBs is both a blessing and a curse |
17:42 |
cr_lf |
I hear you... |
17:42 |
cjb |
.. mostly a blessing :) |
17:42 |
skierpage |
mcha, cr_lf I notice in Tests/Activity/Paint many of the things to test are generic. There are probably pages that imply what a good Activity should do... |
17:42 |
cjl |
HIG? |
17:42 |
mchua |
skierpage: Yes, right now a lot of Activity tests are really Generic Activity Smoke Tests. |
17:42 |
|
(This has to change.) |
17:43 |
|
cjl: HIG is for how they should be designed, but ja |
17:43 |
cjl |
thinks they shoud be designed to work, but that's just me. . . |
17:44 |
cr_lf |
following on from that we could say e.g. rotation is a given therefore the activity is 'broken' unless it handles it cleanly... which gives us the same result in the end... |
17:45 |
mchua |
Let's see. |
17:45 |
skierpage |
cjl, mchua So have a page [[General activity smoke testing]], each Tests/Activity test case refers to it, and that generic page can have discussion of things like screen rotation. |
17:45 |
mchua |
skierpage + 1 |
17:45 |
cjl |
cr_lf: maybe there are jsut some general laptop features tha should be tested in all activities (screen rotation, resume from suspend, etc.) |
17:46 |
|
yeah, what skierpage said. |
17:47 |
mchua |
I'd love someone to take on the "Tell Us How To Smoke Test Activities" project |
17:47 |
skierpage |
Is there [[general activity collaboration testing]] as well as [[general activity testing]] ? |
17:47 |
mchua |
skierpage: nope, collaboration is a whole different beast - I'd like to test them in isolation for this round |
17:47 |
|
to give the sugar devs a bit more time to sort out how collab should actually work |
17:47 |
|
if that's ok |
17:48 |
|
I'd love somebody to come back in 2 weeks and basically tell us "This is how you're going to smoke test Activities!" |
17:48 |
|
and somebody else to come back in 2 weeks and say "And this is how you're going to report results!" |
17:49 |
adricnet |
On the other side of the planet from your damn cool Google spread sheet is Mozilla's Litmus. It's worth a look if only to see what they are doing, even if we don't want/need to set up an install of it. |
17:49 |
mchua |
cr_lf, skierpage - would you like to lead the charge on smoke testing / results reporting respectively? |
17:49 |
|
it sounds like you have way better ideas for implementation than I can think of right now |
17:49 |
|
is slightly burnt out with g1g1 |
17:49 |
adricnet |
Litmus has clicky buttons and collects data cheerfully. |
17:50 |
skierpage |
adricnet, link ? |
17:50 |
mchua |
adricnet: how would you feel about a 2-week-long "let's install and play with as many fun test tools as we can find" party? ;) |
17:51 |
adricnet |
#LINK https://litmus.mozilla.org/index.cgi |
17:51 |
tabitha_ |
by the next G1G1 we should all know exactly how to do it |
17:51 |
mchua |
adricnet: as in, "between now and our next meeting in 2 weeks, your mission, if you so accept, is to Play With Cool Toys and tell us which ones are truly shiny" (can get you serverspace if needed, but I think you already have that through VIG) |
17:51 |
|
tabitha_: how to do Activity testing? |
17:51 |
|
I sure hope so. :) |
17:52 |
adricnet |
mchua: Uh.. that sounds awesome, but not this fortnight :/ |
17:52 |
mchua |
adricnet: no worries. |
17:52 |
skierpage |
mchua, cr_lf : Add action item for "General activity testing" , and I will hunt down existing wiki pages (before someone creates a new page) |
17:52 |
adricnet |
Still I'll see what damage I can inflict this weekend in any slack time |
17:52 |
mchua |
skierpage: so noted, and will do. Thank you! |
17:52 |
adricnet |
I previously tried to install and run litmus and was defeated, sigh. |
17:52 |
mchua |
adricnet: super awesome extra bonus if you can, but I understand you've got a lot on your plate |
17:52 |
|
adricnet: maybe we can ask mozilla's devs to give us a tour |
17:53 |
|
we're running up close to our 1hr time |
17:54 |
skierpage |
*7 sugarbot ?? |
17:54 |
mchua |
tabitha_, aly: how do things look in welly this week? this is a pretty crazy time for US testers, Thanksgiving holiday is coming up so we're all hosed... are you folks busy, or might we persuade you to tell us what to do in 2 weeks' time? ;) |
17:54 |
|
skierpage: garycmartin reported back, no news this week, will try for next time |
17:54 |
|
skierpage: so * 7 sugarbot is done |
17:55 |
cr_lf |
sorry... work intruded... mchua: happy to have a poke through the wiki etc. to flesh something out... |
17:55 |
|
needs an 'advanced wiki wrangling' course... |
17:55 |
mchua |
cr_lf: Awesome. Thanks! I'll get in touch with you over email, but basically you are Now In Charge of telling us how we should do smoke testing of Activities. |
17:55 |
cr_lf |
urk... |
17:55 |
skierpage |
cr_lf 1. assume the info is there 2. assume that you won't find it until *after* you create a new page :-) |
17:55 |
aly |
the usual saturday testing session is happening, but its usually a lottery seeing how many people turn up |
17:55 |
tabitha_ |
WLG Testers will get together tomorrow as usual and results will follow. We can review that spreadsheet and send you our thoughts. |
17:56 |
|
lol - jinx Alastair! |
17:56 |
cr_lf |
skierpage: :-) |
17:56 |
mchua |
skierpage: actually, should the welly testers be telling us what data we should collect? |
17:56 |
skierpage |
tabitha_ thanks. Have you been using the [Add another] test result button in existing test cases to report results? |
17:56 |
mchua |
hey Caryl ! |
17:56 |
|
we're about wrapping up in here right now |
17:57 |
skierpage |
mchua for sure. How are "welly testers" communicating currently? |
17:57 |
tabitha_ |
No, we are a bit scared of changing stuff online so we just started to play with our own wiki page - bit of confidence building - and will now start using the test case pages. We do however all feel confident searching and then reporting bugs we find. |
17:57 |
mchua |
skierpage: tabitha_ and aly and email, mostly - tabitha_, aly? :) |
17:58 |
|
actually, skierpage, tabitha_, aly - maybe we should continue this on email tonight so we don't run too late into the NZ lunch break |
17:58 |
tabitha_ |
We have email distrib list. I have suggested to all WLG testers that they subscribe to the community testing mail as well. |
17:58 |
|
mchua - happy to email |
17:58 |
skierpage |
"community testing mail" == testing lists.laptop.org ? |
17:58 |
mchua |
skierpage: Yup. |
17:59 |
|
Pretty low traffic so far, but it might be getting busier. |
17:59 |
|
Anyway, if we're taking that to email (and cr_lf, I'll catch up with you later as well on smoke tests) |
17:59 |
tabitha_ |
we find it useful |
17:59 |
mchua |
I think that's it for tonight |
17:59 |
tabitha_ |
Thanks Mel |
17:59 |
skierpage |
tabitha_ feel free to use spreadsheet and/or test results inside test cases. Feedback on either from real testers would be awesome. |
17:59 |
mchua |
any last minutes? |
17:59 |
sheba |
thanks mel!! |
17:59 |
mchua |
sheba: thanks for coming! |
18:00 |
|
skierpage: we need to give you a bigger portion of the floor next week :) |
18:00 |
|
last minute calls? |
18:00 |
|
going... |
18:00 |
skierpage |
what's that #AGREE thing someone did? |
18:00 |
mchua |
goin... |
18:00 |
|
skierpage: #AGREE what you agreed on |
18:01 |
|
...going... |
18:01 |
|
#endmeeting |