cosbench changes to support mode==read or mode==mix

[tdeneau]

cosbench: use prepare workstage if mode==read
cosbench: support mix mode
Addresses issues #82

Signed-off-by: Tom Deneau [email protected]

Note: The mixed mode support does not give the user much configurability.
If config.mode is set to mix, we simply treat the "ratio" field as the read ratio,
and add a work operation for write with write_ratio = 100-read_ratio.
Also both read and write cover identical ranges for objects and containers.

[zhouyuan]

Hi @tdeneau
Here's one example for mixed r/w workloads.
https://gist.github.com/zhouyuan/e82fe81b5a64de48221c

Maybe we could create one more template for the "mixed workload", and leave the default template for pure r/w workload? That might be more simpler.

[tdeneau]

Yuan --

I agree that we should allow additional fields to the .yaml file which would
let the user more flexibly configure what containers and object ranges the
mixed reader operation and writer operation should cover like you had
in your example .xml file linked below. For simplicity, the PR I
proposed just always had both reader and writer covering identical ranges
of containers and objects.

I was hesitant to create a whole new template for mixed workloads mainly
because it would share so much in common with the pure r/w template,
so it would have less duplication if we just extended the pure r/w template.

Actually the original template in master does not support a read workload
only a pure write workload so a pure read workload would need an extension
or new template for this as well.

My PR just explicitly added the necessary sub-dictionaries to the template object
after the original template was returned, but maybe a cleaner way would be to
have choose_template return different templates which inherit from the original template.
(which would still avoid the unnecessary duplication).

Also, I'm not sure the end user (.yaml file creator) needs to be aware
of the templates.

-- Tom Deneau

From: Yuan [mailto:[email protected]]
Sent: Sunday, February 14, 2016 7:03 PM
To: ceph/cbt
Cc: Deneau, Tom
Subject: Re: [cbt] cosbench changes to support mode==read or mode==mix (#83)

Hi @tdeneau
Here's one example for mixed r/w workloads.
https://gist.github.com/zhouyuan/e82fe81b5a64de48221c

Maybe we could create one more template for the "mixed workload", and leave the default template for pure r/w workload? That might be more simpler.
—
Reply to this email directly or view it on GitHub.

[xuechendi]

@markhpc , I tested the PR on my cbt env, the codes work well for me, :)
One thing just occur to me is that, current cosbench will init and prepare at every test, if it is a large scale test like 1000_1000 or 10k_10k object test, the prepare stage will be very slow. I am thinking to add a checkup to make sure we only do init&prepare when it is necessary, what do you think?

[markhpc]

On 02/18/2016 07:57 PM, Chendi.Xue wrote:

@markhpc https://github.com/markhpc , I tested the PR on my cbt env,
the codes work well for me, :)
One thing just occur to me is that, current cosbench will init and
prepare at every test, if it is a large scale test like 1000/1000 or
10k/10k object test, the prepare stage will be very slow. I am thinking
to add a checkup to make sure we only do init&prepare when it is
necessary, what do you think?

Sounds good to me. Should we merge this now and add the additional
checks later on?

—
Reply to this email directly or view it on GitHub
#83 (comment).

[xuechendi]

@markhpc , yes, I think we can merge it, I ran a mix-read-write cosbench test, and it looks fine :)
I will start another PR to check and only add init&prepare if needed.

[tdeneau]

@xuechendi , other than a pure write test, what's an example of when prepare would not be required?

[xuechendi]

@tdeneau , if all the obj has been inited and prepared, only need the 'main' workstage, examples like below:

 <?xml version="1.0" encoding="UTF-8"?>
 <workload name="160-cosbench-read-128KB-100con-100obj-100-400-cosbench" description="read-100">
 <storage type="swift" config="timeout=300000" />
 <auth type="swauth" config="username=cosbench:operator;password=intel2012;url=http://10.10.5.21/auth/v1.0;retry=9" />
 <workflow>
 <workstage name="main">
 <work name="128KB" workers="160" rampup="100" runtime="400" rampdown="30">
 <operation type="read" ratio="100" config="containers=r(1,100);objects=r(1,100);cprefix=128KB-read;sizes=c(128)KB"/>
 </work>
 </workstage>
 </workflow>
 </workload>

[tdeneau]

So in the skip-prepare case the objects would have been left over from some previous benchmark run (and some state saved to indicate that)? I agree it would be good to skip the prepare part if we can.

[xuechendi]

I am thinking use Swift interface to check if containers exist, to record some data under CBT also looks good to me, just I am not sure where is the right place to store data.
The check existence PR: #89

From: tdeneau [mailto:[email protected]]
Sent: Tuesday, February 23, 2016 12:14 AM
To: ceph/cbt [email protected]
Cc: Xue, Chendi [email protected]
Subject: Re: [cbt] cosbench changes to support mode==read or mode==mix (#83)

So in the skip-prepare case the objects would have been left over from some previous benchmark run (and some state saved to indicate that)? I agree it would be good to skip the prepare part if we can.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/83#issuecomment-187248796.