Web   ·   Wiki   ·   Activities   ·   Blog   ·   Lists   ·   Chat   ·   Meeting   ·   Bugs   ·   Git   ·   Translate   ·   Archive   ·   People   ·   Donate

#olpc-meeting, 2009-02-17

Index | Today     Channels | Search | Join

All times shown according to UTC.

Time Nick Message
09:12 mchua (for the log: jlaska had some questions about the semantic mediawiki test case system we use.)
09:13 jlaska yeah, so at this point, I'll I'm familiar with is some of the test case mgmt integration with mediawiki
09:14 s/I'll/all/
09:14 we're looking for similar solutions for tracking test execution for several fedora projects
09:15 mchua Cool. What kind of system do you folks use now?
09:15 Ours uses Semantic MediaWiki and Semantic Forms extensions - it's nice and flexible, but there are some drawbacks to it
09:15 jlaska so far, from what I've seen of the laptop.org mechanism ... it seems like a great balance between collaborative wiki content with a light-weight recording mechanism
09:15 we use wiki now ... we deployed testopia last summer, but had to take it down for licensing conflicts
09:15 mchua I'm not sure if I'd call it lightweight; SMW puts a pretty heavy load on the server, and it's a bit slow.
09:15 Ahhh.
09:16 And it's still clunkier than I'd like it to be. New people have trouble learning how to use it.
09:16 That having been said, it works.
09:16 jlaska at this point ... we don't have a lot of resources ... so looking for a quick off-the-shelf solution
09:16 mchua Okay. What would be most helpful to go through?
09:16 Should I walk you through making a test case and a results page with the system?
09:17 And what it would take to set it up for Fedora?
09:17 jlaska we can go there ... howabout just getting this support enabled with mediawiki?
09:17 is familiar with several test case mgmt systems ... but not so much w/ the semantic integration
09:18 mchua Ah, sure - yeah, there are two extensions you'll have to install. Let me find the links.
09:18 With a warning that both these extensions *will* affect performance on your wiki. They're big.
09:18 jlaska okay, good to know up front
09:18 mchua Semantic MediaWiki (SMW) is here: http://semantic-mediawiki.org/[…]emantic_MediaWiki
09:19 It's useful for different reasons other than testing, it lets you perform and build semantic queries on your wiki (more metadata!)
09:19 And then you'll need Semantic Forms, which is http://www.mediawiki.org/wiki/[…]on:Semantic_Forms.
09:19 After your friendly local sysadmin pops those on your mediawiki instance, you're just about ready to go.
09:19 jlaska ooh nice, like the Forms extension
09:20 mchua Both are super-useful.
09:21 Once you have those extensions (and totally feel free to play around on the OLPC wiki to experiment with them before that happens) then the setting-up of the test case system will take...
09:21 jlaska okay ... starting to make sense now with the first link you sent me in email (http://wiki.laptop.org/go/Semantic_MediaWiki)
09:21 mchua ...eh maybe a day of work, I guess. You have an advantage in that you've already got some results and procedures to start from, I'm guessing.
09:22 jlaska yeah, we have a good bit of wiki tests cases, plans and results now
09:22 but of course storing the results in wiki is just painful
09:22 mchua Yeah.
09:22 jlaska and doesn't lend well to sorting/searching/filtering etc...
09:23 mchua The important pages you'll want to copy-paste source from right away:
09:23 http://wiki.laptop.org/go/Template:Test_case
09:23 (That's the form, of sorts)
09:23 and then http://wiki.laptop.org/go/Form:Test_case is the page that creates a page to fill out that form.
09:24 jlaska nice!
09:24 yeah we have a similar template now ... but don't really have anything to leverage it yet (https://fedoraproject.org/wiki[…]ate:QA/Test_Case)
09:25 mchua http://wiki.laptop.org/go/Template:Test_results as well
09:25 jlaska clicks
09:26 nice ... so all your test results are just wiki pages too?
09:26 mchua ...jeez, this is a bit messy. Hm. It might be worth documenting the setup... I don't think Francesca and Charlie took notes while they was figuring out how to make this.
09:26 Yep.
09:26 Well, the test result pages are semantic queries of the test case data.
09:26 Lemme see if I can find one.
09:26 jlaska wow ... and so the semantic stuff then lets you queries (http://wiki.laptop.org/go/TestResults_8.2.0)
09:27 s/you/you build/
09:27 mchua Yep.
09:27 Beat me to it. :)
09:27 It gets a little ugly and unwieldy - I'd say our system is probably ready for a second overhaul, the semantic stuff wasn't set up quite right
09:28 for instance, that particular page o' queries mixes up all the pass/fail results for all the builds
09:29 Oh - another thing you might run into: SMW isn't instantaneous, so when you make a new form/category for the first time, you'll have to wait 5-10 minutes before a query will show up on the page.
09:29 (That panicked me the first time when I couldn't figure out what my erroneous syntax was. So, just so you know.)
09:30 jlaska mchua: sorry, laptop X just hung ... back
09:31 ah ... so there is some batch processing that semantic does to build out pages?
09:31 mchua yup.
09:32 Other things you'll have to work with: if you take a look at a test case like http://wiki.laptop.org/go/Test[…]etworkConnections with "edit with form"
09:32 you'll see things like drop-downs and checkboxes
09:32 jlaska clicks
09:32 mchua in order to set those, you make property pages
09:32 like this: http://wiki.laptop.org/go/Property:Devel_status
09:32 that gives you the values in the dropdown.
09:33 jlaska oh interesting
09:33 so that's the Semantic:Form stuff
09:34 mchua Yep.
09:34 jlaska clever
09:34 mchua This is all pretty haphazard... are there any other things you see in the system (or want to do with yours) that I haven't shown you yet?
09:34 jlaska okay ... can we talk about http://wiki.laptop.org/go/Test[…]etworkConnections for a bit?
09:34 mchua Sure thing.
09:35 (I'd love to help with the implementation of this on your system, btw, to get a clean documentation of how to set this up nicely.)
09:35 jlaska that'd be a big help ... I might have to take you up on that ;)
09:36 so the top of that NetworkConnections page makes sense ... standard test case/procedure document
09:36 it appears that the Semantic stuff adds some information to that page
09:37 mchua Yep, for instance the "How to report results" link is automatic at the top
09:37 jlaska that's part of the Template:
09:37 ?
09:37 mchua Yep, part of http://wiki.laptop.org/go/Template:Test_case
09:38 (view source on that one)
09:38 (er, edit)
09:38 it also gives you the Bright Green Frame of Aesthetic Interestingness
09:39 jlaska heh, I like it :D
09:39 so howabout the section "Facts about Tests/SmokeTest/NetworkConnections"
09:41 mchua That... would be the "We can't figure out how to get SMW to stop spewing that on the page" section.
09:41 jlaska ha!
09:41 same for the test results on that page too?
09:41 mchua Click on one of the magnifying glasses, though.
09:42 That's a bit useful, shows you how to query for just particular semantic tags.
09:42 jlaska yeah it takes you to a Special: page
09:42 ah okay
09:42 mchua Oh, no, we tell the test results to show up there.
09:42 That's how people report results.
09:42 jlaska ah okay
09:42 mchua (http://wiki.laptop.org/go/Reporting_test_results)
09:42 Again, it gets somewhat unwieldy when you scale to having lots of test results.
09:43 But for now, it works. I'm not sure what the next-step solution is yet.
09:43 jlaska there we go ... http://wiki.laptop.org/go/Test_cases_8.2.0  <--- was looking for your "test plan"
09:43 yeah, so can you spend a moment outlining things it does well and things that need improving etc ...
09:43 mchua Sure. Good things first:
09:44 It's very flexible.
09:44 Built into the wiki that we already use.
09:44 It's easy to include plaintext notes that don't fit into the forms, and comments, because of that.
09:44 jlaska nods
09:44 mchua No administrative rights are needed to use or tweak it; a regular user can improve a form.
09:44 jlaska just normal wiki login?
09:44 mchua Yep.
09:45 jlaska nice
09:45 mchua The forms extension makes it easier to contribute results than editing wiki text would make it, I think, though I have no quantitative evidence to back that up.
09:45 jlaska it sounds like the Form extension is handy by itself (just for providing structured wiki input)
09:45 mchua I'd definitely say so.
09:45 jlaska well, that's matches our experiences
09:45 manually editing tables of results is just painful
09:45 and too easy to mess up the table formattin
09:46 one example ... https://fedoraproject.org/wiki[…]tall_Test_Results
09:46 mchua It also gives you *some* sort of structure for test results - the forms with checkboxes and dropdowns - so there's something consistent to query against.
09:46 jlaska right
09:46 mchua Ooh, beautiful table. But I totally hear you on the table syntax.
09:47 Being able to build semantic queries for test results is *awesome.*
09:47 I can do things on the spur of the moment like ask "Show me all the test results for the Browse Activity"
09:47 jlaska how do you do that?
09:47 mchua or "Show me all the Activities that passed test X for these 3 builds"
09:48 You build a query - it'll look something like... one sec....
09:48 jlaska: check out http://wiki.laptop.org/index.p[…]on=edit&section=9
09:48 if you see   [[Category:Test cases]] [[Test category::SmokeTest]]
09:49 that's how you say "give me the intersection of those categories"
09:49 jlaska double :: huh?
09:49 mchua and then the   |?text_goes_here stuff is the data you want to display for the things that match those criteria
09:49 :: == Property
09:50 as in http://wiki.laptop.org/go/Property:Devel_status
09:50 jlaska gotcha
09:50 {{#ask: [[Category:Test cases]] [[Test category::Network]] |?Build stream |?Build number |?Trac bug number |?PassFail |link=subject |default=Nothing in [[:Category:Test cases]] found with [[Test category::Network]]?!
09:50 }}
09:50 errgh ... didn't format whitespace well :(
09:51 btw ... the table formatting makes itm uch easier to grok (for me) ... http://wiki.laptop.org/go/Test_cases_8.2.1
09:52 mchua ooh, that's good to know. easier to grok compared to what?
09:52 jlaska http://wiki.laptop.org/go/Test_cases_8.2.0
09:52 the
09:53 colors on the test case tables just are easier for my cave man brain to process ;)
09:54 "Note! The values in the different columns are unrelated. Just because a build and "Fail" are on the same row doesn't mean that's the failing test result. You have to look at the original test case to see what values are from the same test result. "
09:54 hrmm, rats
09:54 mchua Yeah, that would be the "we didn't design the semantic parts well enough the first time" bit I mentioned last time.
09:55 Avoid Our Mistakes!
09:55 jlaska ah okay, sorry I derailed that discussion ... want to continue the pros/cons ?
09:55 likes the built-in table sorting
09:57 mchua no worries. Yeah, the table class = "sortable" is my new best friend.
09:57 (It's automatically built into recent versions of mediawiki, no extensions needed.)
09:58 jlaska gotcha ... another good thing to note
09:58 mchua Soooo, good things, good things... hm...
09:59 That we have a test case system at all, I suppose. I think I've covered most of my happy pluses. Now for complaining.
09:59 it's big.
09:59 It's slow. The extensions are pretty heavy loaders.
09:59 It is, as you've seen, not the most intuitive thing in the world to setup and use.
09:59 SMW syntax has a learning curve.
10:00 Having Skierpage around (he's very good with SMW) helped us a lot; you might try to get someone from the SMW community to hang out while you're starting with this.
10:00 jlaska do the Form's help insulate users from having to know SMW syntax?
10:00 mchua You end up with bajillions of little separate pages for all of your results.
10:00 The forms do help, for really beginning users.
10:01 And for non-beginning users too, I suppose.
10:01 It just feels like the "friendly wrapper" they put around "scary scary syntax" is very thin and brittle
10:01 i.e. it's easy to click on the wrong thing and go "YAAAH! What is this {{== stuff?"
10:01 jlaska heh, okay :)
10:01 mchua and if you want to edit forms so that different things are easy, *that* process is kinda scary
10:02 jlaska if you had to do it all over again ... where would you start?
10:02 mchua Propagating changes is difficult, too.
10:03 (Always the downside of flexibility, I think.)
10:03 To do it all over again:
10:03 I might start with a separate wiki just for test results.
10:03 That way the main wiki wouldn't be affected by the load of the extensionss.
10:03 (That's a tradeoff I'm not sure of, but I'd at least try it a second time.)
10:03 jlaska so test cases and test plans would be in the main ... but the "[+]" link to add results would route to a test result wiki?
10:04 mchua no, all of it would be in a test results wiki.
10:04 jlaska gotcha
10:04 mchua A separate, clean mediawiki instance with just those two extensions enabled.
10:04 (It's not a dealbreaker to have it all in one, though, and convenient for other reasons.)
10:05 I would write a test workflow before starting to implement the system, so it'd be less haphazard.
10:05 So, have (on static wiki pages or something) things like test results reporting forms already designed out
10:05 I'd have usage instructions - we have some of these, like how to report results
10:05 but I'd also have instructions on how to design new forms in a clean way.
10:06 I would *not* use subpages for test categories. That's a big one.
10:06 jlaska so trying to iron out the process kinks before
10:06 mchua Yeah, process first and then implementation later.
10:06 jlaska oh ... oops, we've got sub categories out the wazzooo
10:06 https://fedoraproject.org/wiki[…]tegory:Test_Cases
10:06 mchua I think you'll have a leg up on us for that, since you already have QA whereas we were mostly starting from scratch
10:07 *grin* if subcategories work for you, awesome.
10:07 jlaska heh, well, I like how they organize things, but I don't know if I've used them as much as you have yet
10:07 mchua It just drives me batty to see things like Tests/Network/100XOs/DLinkRouter/While​StandingOnOneFoot/WearingYellowShirts
10:08 jlaska ah yeah ... we stopped doing that
10:08 but I do use [[Category:bla]] which is a subcategory of [[Category:foo]]
10:08 mchua because when you're trying to add a new test, you're going "But if I'm standing on two feet... and still wearing a yellow shirt..."
10:08 jlaska actually "stopped" ... limited
10:08 mchua nods
10:09 jlaska okay that's good, matches our experience on the "namespace" of the test cases
10:09 mchua Instead of that, I'd use semantic categories to tag test cases.
10:09 I mean, we have them anyway.
10:09 jlaska still has to migrate old style tests over to the QA: namespace
10:09 mchua Why not take advantage?
10:09 Whee, wiki migration.
10:09 jlaska heh .. .that's what we've started to do with our test cases ... https://fedoraproject.org/wiki[…]taller_Test_Cases
10:09 mchua Yeah, subcategories are *way* better than subpages
10:09 Nice!
10:10 jlaska gotcha
10:10 mchua Then you can also do those queries, like "give me all test cases related to networking and f11" instead of /This/Awful/Long/Thing/Nobody/Remembers
10:10 jlaska heh, true
10:11 mchua: this is great stuff!  Are there any other big gotchas you'd want ot address if you got to do it all over?
10:11 mchua I would have and designate a few people to be Keepers of The Test System
10:11 and put their names up on the howto instructions page.
10:11 jlaska a select gropu to make sure content is landing in the right places + format?
10:11 mchua Yep.
10:11 I would definitely have a more formal rollout of the system.
10:12 Er, in that I would have a rollout.
10:12 jlaska hehe
10:12 mchua We basically set it up and started using it internally
10:12 jlaska heh, sometimes that's the fastest way to move
10:12 mchua volunteers would come in and we'd say "use the system!" and they'd say "what system?"
10:12 jlaska yeah
10:12 is all too familiar
10:13 would you ever consider a stand-alone web app for test mgmt?
10:13 something like a testopia, litmus, testlink etc...
10:13 mchua I definitely would.
10:13 In fact, that's something I've been thinking about trying out / convincing someone else to
10:13 jlaska not that it solves any/all process issues ... but just tossing it out there
10:14 mchua I'm not super familiar with what's out there for that, though
10:14 It's likely to be faster and easier to use for this particular purpose
10:15 jlaska yeah, it also requires a bit more skills on the admin side of things.  I guess wiki is "self-administered" (to a point) and really *great* for content
10:15 mchua And more scaleable. But 'till we knew what our process was, there wasn't much point in trying those out, I think
10:15 jlaska I agre
10:15 e
10:15 mchua I think of SMW as a lovely prototyping tool.
10:15 jlaska yeah, that's a great way to think of it
10:15 mchua: okay ... well I appreciate your time and this information is really helpful!
10:15 mchua: I'm going to process/digest for a bit ... and go over the links you've noted
10:16 mchua no problem! would love to see it if you get it going; you'll probably get a better gen2 system up that we can then learn from :)
10:16 jlaska yeah, I'll certainly need your expertise depending on how we move with this
10:16 mchua can you do me a favor and email the logs from this convo to testing@lists.laptop.org with any Really Big High Level Notes/Revelations?
10:16 (I'll send you a link to the logs in a sec)
10:17 jlaska of course
10:17 mchua it'd make a nice resource to have out there for the folks who haven't gotten into SMW yet
10:17 thanks!
10:17 #endmeeting

Index | Today     Channels | Search | Join

Powered by ilbot/Modified.
Webmaster