| Time |
Nick |
Message |
| 09:12 |
mchua |
(for the log: jlaska had some questions about the semantic mediawiki test case system we use.) |
| 09:13 |
jlaska |
yeah, so at this point, I'll I'm familiar with is some of the test case mgmt integration with mediawiki |
| 09:14 |
|
s/I'll/all/ |
| 09:14 |
|
we're looking for similar solutions for tracking test execution for several fedora projects |
| 09:15 |
mchua |
Cool. What kind of system do you folks use now? |
| 09:15 |
|
Ours uses Semantic MediaWiki and Semantic Forms extensions - it's nice and flexible, but there are some drawbacks to it |
| 09:15 |
jlaska |
so far, from what I've seen of the laptop.org mechanism ... it seems like a great balance between collaborative wiki content with a light-weight recording mechanism |
| 09:15 |
|
we use wiki now ... we deployed testopia last summer, but had to take it down for licensing conflicts |
| 09:15 |
mchua |
I'm not sure if I'd call it lightweight; SMW puts a pretty heavy load on the server, and it's a bit slow. |
| 09:15 |
|
Ahhh. |
| 09:16 |
|
And it's still clunkier than I'd like it to be. New people have trouble learning how to use it. |
| 09:16 |
|
That having been said, it works. |
| 09:16 |
jlaska |
at this point ... we don't have a lot of resources ... so looking for a quick off-the-shelf solution |
| 09:16 |
mchua |
Okay. What would be most helpful to go through? |
| 09:16 |
|
Should I walk you through making a test case and a results page with the system? |
| 09:17 |
|
And what it would take to set it up for Fedora? |
| 09:17 |
jlaska |
we can go there ... howabout just getting this support enabled with mediawiki? |
| 09:17 |
|
is familiar with several test case mgmt systems ... but not so much w/ the semantic integration |
| 09:18 |
mchua |
Ah, sure - yeah, there are two extensions you'll have to install. Let me find the links. |
| 09:18 |
|
With a warning that both these extensions *will* affect performance on your wiki. They're big. |
| 09:18 |
jlaska |
okay, good to know up front |
| 09:18 |
mchua |
Semantic MediaWiki (SMW) is here: http://semantic-mediawiki.org/[…]emantic_MediaWiki |
| 09:19 |
|
It's useful for different reasons other than testing, it lets you perform and build semantic queries on your wiki (more metadata!) |
| 09:19 |
|
And then you'll need Semantic Forms, which is http://www.mediawiki.org/wiki/[…]on:Semantic_Forms. |
| 09:19 |
|
After your friendly local sysadmin pops those on your mediawiki instance, you're just about ready to go. |
| 09:19 |
jlaska |
ooh nice, like the Forms extension |
| 09:20 |
mchua |
Both are super-useful. |
| 09:21 |
|
Once you have those extensions (and totally feel free to play around on the OLPC wiki to experiment with them before that happens) then the setting-up of the test case system will take... |
| 09:21 |
jlaska |
okay ... starting to make sense now with the first link you sent me in email (http://wiki.laptop.org/go/Semantic_MediaWiki) |
| 09:21 |
mchua |
...eh maybe a day of work, I guess. You have an advantage in that you've already got some results and procedures to start from, I'm guessing. |
| 09:22 |
jlaska |
yeah, we have a good bit of wiki tests cases, plans and results now |
| 09:22 |
|
but of course storing the results in wiki is just painful |
| 09:22 |
mchua |
Yeah. |
| 09:22 |
jlaska |
and doesn't lend well to sorting/searching/filtering etc... |
| 09:23 |
mchua |
The important pages you'll want to copy-paste source from right away: |
| 09:23 |
|
http://wiki.laptop.org/go/Template:Test_case |
| 09:23 |
|
(That's the form, of sorts) |
| 09:23 |
|
and then http://wiki.laptop.org/go/Form:Test_case is the page that creates a page to fill out that form. |
| 09:24 |
jlaska |
nice! |
| 09:24 |
|
yeah we have a similar template now ... but don't really have anything to leverage it yet (https://fedoraproject.org/wiki[…]ate:QA/Test_Case) |
| 09:25 |
mchua |
http://wiki.laptop.org/go/Template:Test_results as well |
| 09:25 |
jlaska |
clicks |
| 09:26 |
|
nice ... so all your test results are just wiki pages too? |
| 09:26 |
mchua |
...jeez, this is a bit messy. Hm. It might be worth documenting the setup... I don't think Francesca and Charlie took notes while they was figuring out how to make this. |
| 09:26 |
|
Yep. |
| 09:26 |
|
Well, the test result pages are semantic queries of the test case data. |
| 09:26 |
|
Lemme see if I can find one. |
| 09:26 |
jlaska |
wow ... and so the semantic stuff then lets you queries (http://wiki.laptop.org/go/TestResults_8.2.0) |
| 09:27 |
|
s/you/you build/ |
| 09:27 |
mchua |
Yep. |
| 09:27 |
|
Beat me to it. :) |
| 09:27 |
|
It gets a little ugly and unwieldy - I'd say our system is probably ready for a second overhaul, the semantic stuff wasn't set up quite right |
| 09:28 |
|
for instance, that particular page o' queries mixes up all the pass/fail results for all the builds |
| 09:29 |
|
Oh - another thing you might run into: SMW isn't instantaneous, so when you make a new form/category for the first time, you'll have to wait 5-10 minutes before a query will show up on the page. |
| 09:29 |
|
(That panicked me the first time when I couldn't figure out what my erroneous syntax was. So, just so you know.) |
| 09:30 |
jlaska |
mchua: sorry, laptop X just hung ... back |
| 09:31 |
|
ah ... so there is some batch processing that semantic does to build out pages? |
| 09:31 |
mchua |
yup. |
| 09:32 |
|
Other things you'll have to work with: if you take a look at a test case like http://wiki.laptop.org/go/Test[…]etworkConnections with "edit with form" |
| 09:32 |
|
you'll see things like drop-downs and checkboxes |
| 09:32 |
jlaska |
clicks |
| 09:32 |
mchua |
in order to set those, you make property pages |
| 09:32 |
|
like this: http://wiki.laptop.org/go/Property:Devel_status |
| 09:32 |
|
that gives you the values in the dropdown. |
| 09:33 |
jlaska |
oh interesting |
| 09:33 |
|
so that's the Semantic:Form stuff |
| 09:34 |
mchua |
Yep. |
| 09:34 |
jlaska |
clever |
| 09:34 |
mchua |
This is all pretty haphazard... are there any other things you see in the system (or want to do with yours) that I haven't shown you yet? |
| 09:34 |
jlaska |
okay ... can we talk about http://wiki.laptop.org/go/Test[…]etworkConnections for a bit? |
| 09:34 |
mchua |
Sure thing. |
| 09:35 |
|
(I'd love to help with the implementation of this on your system, btw, to get a clean documentation of how to set this up nicely.) |
| 09:35 |
jlaska |
that'd be a big help ... I might have to take you up on that ;) |
| 09:36 |
|
so the top of that NetworkConnections page makes sense ... standard test case/procedure document |
| 09:36 |
|
it appears that the Semantic stuff adds some information to that page |
| 09:37 |
mchua |
Yep, for instance the "How to report results" link is automatic at the top |
| 09:37 |
jlaska |
that's part of the Template: |
| 09:37 |
|
? |
| 09:37 |
mchua |
Yep, part of http://wiki.laptop.org/go/Template:Test_case |
| 09:38 |
|
(view source on that one) |
| 09:38 |
|
(er, edit) |
| 09:38 |
|
it also gives you the Bright Green Frame of Aesthetic Interestingness |
| 09:39 |
jlaska |
heh, I like it :D |
| 09:39 |
|
so howabout the section "Facts about Tests/SmokeTest/NetworkConnections" |
| 09:41 |
mchua |
That... would be the "We can't figure out how to get SMW to stop spewing that on the page" section. |
| 09:41 |
jlaska |
ha! |
| 09:41 |
|
same for the test results on that page too? |
| 09:41 |
mchua |
Click on one of the magnifying glasses, though. |
| 09:42 |
|
That's a bit useful, shows you how to query for just particular semantic tags. |
| 09:42 |
jlaska |
yeah it takes you to a Special: page |
| 09:42 |
|
ah okay |
| 09:42 |
mchua |
Oh, no, we tell the test results to show up there. |
| 09:42 |
|
That's how people report results. |
| 09:42 |
jlaska |
ah okay |
| 09:42 |
mchua |
(http://wiki.laptop.org/go/Reporting_test_results) |
| 09:42 |
|
Again, it gets somewhat unwieldy when you scale to having lots of test results. |
| 09:43 |
|
But for now, it works. I'm not sure what the next-step solution is yet. |
| 09:43 |
jlaska |
there we go ... http://wiki.laptop.org/go/Test_cases_8.2.0 <--- was looking for your "test plan" |
| 09:43 |
|
yeah, so can you spend a moment outlining things it does well and things that need improving etc ... |
| 09:43 |
mchua |
Sure. Good things first: |
| 09:44 |
|
It's very flexible. |
| 09:44 |
|
Built into the wiki that we already use. |
| 09:44 |
|
It's easy to include plaintext notes that don't fit into the forms, and comments, because of that. |
| 09:44 |
jlaska |
nods |
| 09:44 |
mchua |
No administrative rights are needed to use or tweak it; a regular user can improve a form. |
| 09:44 |
jlaska |
just normal wiki login? |
| 09:44 |
mchua |
Yep. |
| 09:45 |
jlaska |
nice |
| 09:45 |
mchua |
The forms extension makes it easier to contribute results than editing wiki text would make it, I think, though I have no quantitative evidence to back that up. |
| 09:45 |
jlaska |
it sounds like the Form extension is handy by itself (just for providing structured wiki input) |
| 09:45 |
mchua |
I'd definitely say so. |
| 09:45 |
jlaska |
well, that's matches our experiences |
| 09:45 |
|
manually editing tables of results is just painful |
| 09:45 |
|
and too easy to mess up the table formattin |
| 09:46 |
|
one example ... https://fedoraproject.org/wiki[…]tall_Test_Results |
| 09:46 |
mchua |
It also gives you *some* sort of structure for test results - the forms with checkboxes and dropdowns - so there's something consistent to query against. |
| 09:46 |
jlaska |
right |
| 09:46 |
mchua |
Ooh, beautiful table. But I totally hear you on the table syntax. |
| 09:47 |
|
Being able to build semantic queries for test results is *awesome.* |
| 09:47 |
|
I can do things on the spur of the moment like ask "Show me all the test results for the Browse Activity" |
| 09:47 |
jlaska |
how do you do that? |
| 09:47 |
mchua |
or "Show me all the Activities that passed test X for these 3 builds" |
| 09:48 |
|
You build a query - it'll look something like... one sec.... |
| 09:48 |
|
jlaska: check out http://wiki.laptop.org/index.p[…]on=edit§ion=9 |
| 09:48 |
|
if you see [[Category:Test cases]] [[Test category::SmokeTest]] |
| 09:49 |
|
that's how you say "give me the intersection of those categories" |
| 09:49 |
jlaska |
double :: huh? |
| 09:49 |
mchua |
and then the |?text_goes_here stuff is the data you want to display for the things that match those criteria |
| 09:49 |
|
:: == Property |
| 09:50 |
|
as in http://wiki.laptop.org/go/Property:Devel_status |
| 09:50 |
jlaska |
gotcha |
| 09:50 |
|
{{#ask: [[Category:Test cases]] [[Test category::Network]] |?Build stream |?Build number |?Trac bug number |?PassFail |link=subject |default=Nothing in [[:Category:Test cases]] found with [[Test category::Network]]?! |
| 09:50 |
|
}} |
| 09:50 |
|
errgh ... didn't format whitespace well :( |
| 09:51 |
|
btw ... the table formatting makes itm uch easier to grok (for me) ... http://wiki.laptop.org/go/Test_cases_8.2.1 |
| 09:52 |
mchua |
ooh, that's good to know. easier to grok compared to what? |
| 09:52 |
jlaska |
http://wiki.laptop.org/go/Test_cases_8.2.0 |
| 09:52 |
|
the |
| 09:53 |
|
colors on the test case tables just are easier for my cave man brain to process ;) |
| 09:54 |
|
"Note! The values in the different columns are unrelated. Just because a build and "Fail" are on the same row doesn't mean that's the failing test result. You have to look at the original test case to see what values are from the same test result. " |
| 09:54 |
|
hrmm, rats |
| 09:54 |
mchua |
Yeah, that would be the "we didn't design the semantic parts well enough the first time" bit I mentioned last time. |
| 09:55 |
|
Avoid Our Mistakes! |
| 09:55 |
jlaska |
ah okay, sorry I derailed that discussion ... want to continue the pros/cons ? |
| 09:55 |
|
likes the built-in table sorting |
| 09:57 |
mchua |
no worries. Yeah, the table class = "sortable" is my new best friend. |
| 09:57 |
|
(It's automatically built into recent versions of mediawiki, no extensions needed.) |
| 09:58 |
jlaska |
gotcha ... another good thing to note |
| 09:58 |
mchua |
Soooo, good things, good things... hm... |
| 09:59 |
|
That we have a test case system at all, I suppose. I think I've covered most of my happy pluses. Now for complaining. |
| 09:59 |
|
it's big. |
| 09:59 |
|
It's slow. The extensions are pretty heavy loaders. |
| 09:59 |
|
It is, as you've seen, not the most intuitive thing in the world to setup and use. |
| 09:59 |
|
SMW syntax has a learning curve. |
| 10:00 |
|
Having Skierpage around (he's very good with SMW) helped us a lot; you might try to get someone from the SMW community to hang out while you're starting with this. |
| 10:00 |
jlaska |
do the Form's help insulate users from having to know SMW syntax? |
| 10:00 |
mchua |
You end up with bajillions of little separate pages for all of your results. |
| 10:00 |
|
The forms do help, for really beginning users. |
| 10:01 |
|
And for non-beginning users too, I suppose. |
| 10:01 |
|
It just feels like the "friendly wrapper" they put around "scary scary syntax" is very thin and brittle |
| 10:01 |
|
i.e. it's easy to click on the wrong thing and go "YAAAH! What is this {{== stuff?" |
| 10:01 |
jlaska |
heh, okay :) |
| 10:01 |
mchua |
and if you want to edit forms so that different things are easy, *that* process is kinda scary |
| 10:02 |
jlaska |
if you had to do it all over again ... where would you start? |
| 10:02 |
mchua |
Propagating changes is difficult, too. |
| 10:03 |
|
(Always the downside of flexibility, I think.) |
| 10:03 |
|
To do it all over again: |
| 10:03 |
|
I might start with a separate wiki just for test results. |
| 10:03 |
|
That way the main wiki wouldn't be affected by the load of the extensionss. |
| 10:03 |
|
(That's a tradeoff I'm not sure of, but I'd at least try it a second time.) |
| 10:03 |
jlaska |
so test cases and test plans would be in the main ... but the "[+]" link to add results would route to a test result wiki? |
| 10:04 |
mchua |
no, all of it would be in a test results wiki. |
| 10:04 |
jlaska |
gotcha |
| 10:04 |
mchua |
A separate, clean mediawiki instance with just those two extensions enabled. |
| 10:04 |
|
(It's not a dealbreaker to have it all in one, though, and convenient for other reasons.) |
| 10:05 |
|
I would write a test workflow before starting to implement the system, so it'd be less haphazard. |
| 10:05 |
|
So, have (on static wiki pages or something) things like test results reporting forms already designed out |
| 10:05 |
|
I'd have usage instructions - we have some of these, like how to report results |
| 10:05 |
|
but I'd also have instructions on how to design new forms in a clean way. |
| 10:06 |
|
I would *not* use subpages for test categories. That's a big one. |
| 10:06 |
jlaska |
so trying to iron out the process kinks before |
| 10:06 |
mchua |
Yeah, process first and then implementation later. |
| 10:06 |
jlaska |
oh ... oops, we've got sub categories out the wazzooo |
| 10:06 |
|
https://fedoraproject.org/wiki[…]tegory:Test_Cases |
| 10:06 |
mchua |
I think you'll have a leg up on us for that, since you already have QA whereas we were mostly starting from scratch |
| 10:07 |
|
*grin* if subcategories work for you, awesome. |
| 10:07 |
jlaska |
heh, well, I like how they organize things, but I don't know if I've used them as much as you have yet |
| 10:07 |
mchua |
It just drives me batty to see things like Tests/Network/100XOs/DLinkRouter/WhileStandingOnOneFoot/WearingYellowShirts |
| 10:08 |
jlaska |
ah yeah ... we stopped doing that |
| 10:08 |
|
but I do use [[Category:bla]] which is a subcategory of [[Category:foo]] |
| 10:08 |
mchua |
because when you're trying to add a new test, you're going "But if I'm standing on two feet... and still wearing a yellow shirt..." |
| 10:08 |
jlaska |
actually "stopped" ... limited |
| 10:08 |
mchua |
nods |
| 10:09 |
jlaska |
okay that's good, matches our experience on the "namespace" of the test cases |
| 10:09 |
mchua |
Instead of that, I'd use semantic categories to tag test cases. |
| 10:09 |
|
I mean, we have them anyway. |
| 10:09 |
jlaska |
still has to migrate old style tests over to the QA: namespace |
| 10:09 |
mchua |
Why not take advantage? |
| 10:09 |
|
Whee, wiki migration. |
| 10:09 |
jlaska |
heh .. .that's what we've started to do with our test cases ... https://fedoraproject.org/wiki[…]taller_Test_Cases |
| 10:09 |
mchua |
Yeah, subcategories are *way* better than subpages |
| 10:09 |
|
Nice! |
| 10:10 |
jlaska |
gotcha |
| 10:10 |
mchua |
Then you can also do those queries, like "give me all test cases related to networking and f11" instead of /This/Awful/Long/Thing/Nobody/Remembers |
| 10:10 |
jlaska |
heh, true |
| 10:11 |
|
mchua: this is great stuff! Are there any other big gotchas you'd want ot address if you got to do it all over? |
| 10:11 |
mchua |
I would have and designate a few people to be Keepers of The Test System |
| 10:11 |
|
and put their names up on the howto instructions page. |
| 10:11 |
jlaska |
a select gropu to make sure content is landing in the right places + format? |
| 10:11 |
mchua |
Yep. |
| 10:11 |
|
I would definitely have a more formal rollout of the system. |
| 10:12 |
|
Er, in that I would have a rollout. |
| 10:12 |
jlaska |
hehe |
| 10:12 |
mchua |
We basically set it up and started using it internally |
| 10:12 |
jlaska |
heh, sometimes that's the fastest way to move |
| 10:12 |
mchua |
volunteers would come in and we'd say "use the system!" and they'd say "what system?" |
| 10:12 |
jlaska |
yeah |
| 10:12 |
|
is all too familiar |
| 10:13 |
|
would you ever consider a stand-alone web app for test mgmt? |
| 10:13 |
|
something like a testopia, litmus, testlink etc... |
| 10:13 |
mchua |
I definitely would. |
| 10:13 |
|
In fact, that's something I've been thinking about trying out / convincing someone else to |
| 10:13 |
jlaska |
not that it solves any/all process issues ... but just tossing it out there |
| 10:14 |
mchua |
I'm not super familiar with what's out there for that, though |
| 10:14 |
|
It's likely to be faster and easier to use for this particular purpose |
| 10:15 |
jlaska |
yeah, it also requires a bit more skills on the admin side of things. I guess wiki is "self-administered" (to a point) and really *great* for content |
| 10:15 |
mchua |
And more scaleable. But 'till we knew what our process was, there wasn't much point in trying those out, I think |
| 10:15 |
jlaska |
I agre |
| 10:15 |
|
e |
| 10:15 |
mchua |
I think of SMW as a lovely prototyping tool. |
| 10:15 |
jlaska |
yeah, that's a great way to think of it |
| 10:15 |
|
mchua: okay ... well I appreciate your time and this information is really helpful! |
| 10:15 |
|
mchua: I'm going to process/digest for a bit ... and go over the links you've noted |
| 10:16 |
mchua |
no problem! would love to see it if you get it going; you'll probably get a better gen2 system up that we can then learn from :) |
| 10:16 |
jlaska |
yeah, I'll certainly need your expertise depending on how we move with this |
| 10:16 |
mchua |
can you do me a favor and email the logs from this convo to testing lists.laptop.org with any Really Big High Level Notes/Revelations? |
| 10:16 |
|
(I'll send you a link to the logs in a sec) |
| 10:17 |
jlaska |
of course |
| 10:17 |
mchua |
it'd make a nice resource to have out there for the folks who haven't gotten into SMW yet |
| 10:17 |
|
thanks! |
| 10:17 |
|
#endmeeting |