Dear all,

Please find the proposed response for the phi K* paper below. It is not a great paper. I hope we can make them sweat a little.

If you'd still like me to include your comments then please let me know. Otherwise, I'll send this in tomorrow.

Cheers, Wouter

----------------------------------------------------

Dear authors,

I report on the review of your paper by the LHCb group at Nikhef. We would like to congratulate you with this interesting analysis. We do have some concerns, both on the analysis and on the way it is reported. Please find our comments below.

Cheers, Wouter (with help from Patrick K.)

Analysis comments:

1. In line 131 you explain that your GL training explicitly removes phi. Does that mean that real phi are not your typical background? We would have expected the opposite. If you had trained on real phi's maybe the Bs->phiphi background (L.139) wouldn't be a problem?

In line 140 you mention that you explicitly remove the phiphi background with a veto on the 4 body mass. Did you check that this does not affect the shape of your combinatorial background? (You could expect a dip somewhere.)

2. We find the motivation for your mass model rather weak. As you indicate in the paper the estimated fraction of Bs->KKKpi events is very sensitive to the upper tail of the Bd->KKKpi signal.

a. You use a sum of a CB and a Gaussian, but how do you prove that this Gaussian accurately describes the high mass tail? You explain in line 178b that you fix the gaussian resolution by exploiting a Bd->J/psiK* decays. Why is this sample representative for your signal? We know that phi reconstruction is complicated due to tracks being close: Why would you expect the tails in J/psiK* to have anything to do with those in phiK*? We could not find a study of the Bd->phiK* mass distribution in the analysis note. The fit to simulated Bs->phiK* events (Fig 4 bottom) does not exceed to high enough mass to see if the gauss really describes the KKKpi tail.

b. You assume that the mass shape for Bd and Bs is identical. However, you don't expect them to have exactly the same resolution, because of the difference in mass. How did you account for this?

c. You state that the gaussian accounts for particles that have suffered a hard collision. This is probably the case for a small fraction of the events. However, if you'd look at the distribution of the estimated error on the mass (from the vertex fit) you'll see that you actually don't expect the distribution to be described by a single gaussian. (Rather, you may reduce the non-gaussian behaviour by cutting on the mass error.)

2. We are not convinced about the reported significance. First, to report a significance in excess of 6 sigma, you would either need to do 10^8 toy expriments or somehow rely on extrapolation/approximation (Wilks' theorem). Can you really estimate the significance with three digits precision just from the dLL?

a. We tried to evaluate the significance just from the numbers that you report: yield = 29.9 ± 6.4 -> signal + background = 6.4^2 = 41.0 -> background = 41.0 - 29.9 = 11.1 -> probability for 11.1 events to fluctuate to >=41 events = = int_i=sum_41^\infty TMath::Poison(11.1,i) = 4.7 x 10-8 -> this is "5.5 sigma"

We don't claim that this estimate is any better, but we still wonder how you can get to >6 sigma with this signal yield: even if we assume that 6.4 is somehow a rounded number and start from something like 6.24 (which corresponds to 39 signal+background) we don't get to more than 5.7 sigma significance.

b. In line 197 you report also a number of estimated signal/background events in a signal region. This would in principle allow for a similar computation as above, but you should really also report the /actual/ number of events found in this window (so not just what is extracted by your fit).

If we just use the numbers that are there, assuming most optimistically that the observed number of events is rint(26.4 + 8.2)=35 for 8.2 background events, then the significance is about 5.5 sigma.

c. We are puzzled by your correction of the signficance. You report a significance from a dLL, because, unless you are background dominated, the formula 'Nsig/sigma(Nsig)' is simply not good enough. Yet, when it comes to estimating the effects from background shape uncertainty, you do as if this is exactly the kind of formula that you have applied to compute the significance. Does this really make sense?

As an alternative, may we suggest that you take the background model that gave the smallest signal yield, extract the dLL from that fit, and translate this into a significance, rounding off to the nearest digit. In other words, all you need to say is:

"Taking into account uncertainties in the background model, we estimate the significance of our result to exceed 5.5 [or eventually 6] sigma."

d. Your reported significance is the significance for observing KKKpi in a 14x300 MeV^2 window. You determine that only 89% of that is phiK*. The significance of Bs->phiK* should be determined from the probability of all backgrounds to fluctuate to the yield you see. These backgrounds include the non-resonant component (as this is not part of what you report). How does your significance change if you take that into account?

3. We are puzzled by your analysis of the S-wave fraction. The amplitudes in Bd->phiKpi have been quite extensively studied by the B-factories, for instance in Ref 11. Babar has shown that there are only three amplitudes contributing, namely the S-wave K_0*(1430), the P-wave K_1*(890) and the tensor K_2*(1430). They extract these numbers with a fit that includes both the mass and the angles.

a.) We do find it difficult to believe that one can reliably extract the s-wave fraction from a fit to just the invariant KK, Kpi mass: you'll depend on knowledge of the tails in the mass resolution function and of the B and combinatorial backgrounds that may not be easy to quantify. It seems that you ignore the background and simply convolve the Breit-Wigner for the K* with a gaussian, even though the resolution depends quite strongly on the Kpi or KK invariant mass. In your 2sigma window around the B mass you may have some background left. How do you estimate uncertainties for these effects? Have you validated this procedure on MC?

We think that the result may be more convincing if you include the K* and phi mass distributions and fits for both Bd and Bs.

b.) You assume that the -swave fraction is the same in Bs and Bd decays and assign a 50% error. How do you justify this error? You sort of motivate your strategy by saying that a fit with the S-wave contribution fits better than without in line 225. But how sensitive are you? Maybe a 50% S-wave fraction fits even better. If you really have sensitivity to the s-wave fraction in Bs decays, then why don't you just fit the s-wave fraction in Bs decays the same way you do in Bd?

c.) One would expect that a fit that exploits decay angles is more robust against background and the exact mass model. Why haven't you used such a 2D fit? (Please also explain in the paper!)

d.) Looking at figure 4 in the babar paper, you wonder if the "11+/-2"% KPi s-wave fraction isn't a bit large. (We guestimated 5% from the babar paper.) Are you sure that the 11% is consistent with the babar measurement? If it is more precise, why don't you describe it better such that it can actually be used by others? If it is less precise, why don't you use the babar result instead?

e.) We were wondering the same about the 5% s-wave that you find under the phi. The babar paper is a bit vague about this (they don't actually quote a BF for the KK s-wave) but they do report that in the K* mass window, they have 84+/-19 non resonant KK out of approximately 500 events, which seems considerably more than the 5% that we have. On the other hand, in the J/psiphi analysis, the non-resonant phi contribution in the narrow phi window is definitely less than 1%. How consistent is the 5% with other experiments?

4. In section 7 you report the results from a polarization study of Bs->phiK*.

a. Since the Bd->phiK* polarization has been measured before, we had expected that you would also report polarization results for Bd->phiK*. Did you not actually measure the Bd->phiK* polarization? It would probably be the worlds most precise measurement and furthermore it would give more credibility to the analysis if you can show that on this 'control channel' you are consistent with the WA. In fact, we think that it is mandatory to do so.

b. You estimate the effect of an S-wave by doing a P-wave only fit on toy samples that have an S-wave included. Once you have gone trough the effort of writing down a PDF that includes S-wave, why don't you just fit the S-wave fraction on the data? Naively, this would give you a much better sensitivity to the S-wave fraction than fitting a mass peak. Furthermore, it would remove your dominant systematic uncertainty.

What do you assume for the strong phase difference between P-wave and S-wave in the toy simulations?

c. In line 264 you explain that you 'weigh' the events. Did you weigh with 1/efficiency? (If so, please say this in the text.) The problem with event weighting is that it becomes harder to compute statistical uncertainties, in particular for small samples. How do you correct for this? Did you performs toys to test if your statistical errors are fine? Why did you use weighting rather than including an efficiency model in the fit, which is more common in LHCb?

5. In line 282 you state that you extract a systematic uncertainty by replacing the GL with a set of rectangular cuts. Why would that tell you anything about systematics? If you have used rectangular cuts, would you have checked for biases with a GL? We don't think this needs to be included in the systematics.

Corrections/suggestions for the text -----------------------------

structure: Since the polarization has an effect on your efficiency, and hence on the branching fraction, wouldn't it make more sense to move section 6 behind section 7?

abstract: of 6.06 sigma -> in excess of 6 sigma

2: use template \cp 3: crucial --> important 3: B mesons 4: do you really need NP? In line 8 "would indicate NP" doesn't read very well. 7: SM CP violation -> the SM prediction for CP violation 9: given by -> by a factor. "consequently... scale" is too vague. What scale? 13: $B$-factories. Here and in many places you use a long dash -- when you should be using a hyphen -. q is undefined. 14: K_S : Here and in many other places use the LHCb template for particles. 15: Do Time and Violation really need to be capitalised? 16: Ref.~[4] 17: "more valuable" wrt what? 20: highly -> dominantly (?) 22-23 and many other places. You use rho, omega, K*, phi without stating that you mean rho(780), K*(892), phi(1020)... 25: comparable with, and -> of similar size, or 27-28: We am not sure how I should understand this sentence. Is the comma at the right place? 29, 32: ~ -> about 31: There seems to be a large space in front of [17 32: "quite large" is vague 33: Ref.~[20]. 34: There must be more other B->VV modes if you count higher excitations of phi and K*. 37: No error on 1.3? Without that info we cannot judge if the statement "not compatible" is correct or not. What's the level of incompatibility? 39: "some evidence" is vague 43: into light vector. This sentence is long. Maybe cut after the parenthesis. 48: what do the different errors on the theoretical result mean? 49: which polarization puzzle? The one mentioned in lines 23-26? 73: 3 kHz 81: \bquark-hadron 95 and many other places: It seems to be assumed that kaons and pions are always charged. Please say so. It would make the paper clearer if you always put the charge. 96: the KK -> a KK footnote 2: {miss} -> \text{miss} (and \usepackage{amsmath{ Same for corr. 99: which vertex is this? the phi/K* vertex? or the B vertex? is this the same vertex as for the chi2 cut in line 112? if so, why repoert both? 100: which trajectories? the phi trajectory and the K* trajectory? why don't you just make a cut on the B vertex chi2? 110: remove baryonic. Lambda_b is not the one from the template. 113: how do you compute the 'B direction'? because if you compute it from the B momentum, then it seems completely impossible that the cosine of this angle is not positive. (all particles go within a 300mrad cone in LHCb.) 116: opening -> momentum 117: +/-7 Mev around what? 118: physics -> natural (?) 119: No dash in Geometrical Likelihood 120: No dash before meson 125: "reconstructed from... tracks" what does that mean? why don't you just use the vertex chi2? 136: Boosted Decision Trees should be singular and we have a standard sentence for that. See recent publications. But since you don't use the BDT, why do you report this? bottom page 4: please just insert these footnotes in the text 142: simulation -> simulated 145: Ds: use template 150: DLL_pk -> DLL_pK 151: use \L from template. Also Lambda_c 158: small sample of two combinations -> small contribution from the decays 165: Reference missing for mass difference. 166: "plus a Gaussian" means what? 167: You don't need to define the CB. 170-171: Do you have evidence from simulation for this claim? Our experience is that the tails come from events that have a large mass error due to long extrapolation distances and small opening angles. If your explanation is correct then a Gaussian model is not the best description. You can easily kill these candidates using the mass error. 176: the parameter you fit for is that ratio of the factors of Gaussian and CB (not the ratio of the functions). Table 1 and 191-192: You use Events and Candidates interchangeably. Are they? 191: Either begin a new setnence after 6.4 or make sure the two subsentences have the same subject. 195: why parentheses? 196: "Integration" around what? Lines 202-206: is the formula really needed? Fig2: what's the point of the top plot? There's a parenthesis too much in the label. M and c should be italics. 212: you need to refer her to studies of Bd->phiK* by the B factories and probably report what they found for the non P-wave contribution. 216: form -> from 223: The p_T of the daughters could also come from a different p_T of the B mesons, which could be a much larger effect. We have measured f_d/f_s in bins of P_T. You can refer to that. Eq.1 : f_\dquark, {\rm Acc} (also elsewhere) 240: are discussed in section 8. No comma after 10^-6 242-246: This comment is meaningless here as you have not yet included a systematic error. Please move this to the conclusion. Table 2: to -> used in with -> in the ratio fd/fs is missing from this table. do you actually need this table? 255: "The angle convention is shown..." also add it is described in the caption. Cannot you just define the angles in a formula? 257: That optimizes... what optimizes? And why? 259: period missing. 263: calculation ... depends on -> determination ... is affected by. 265: was -> is. Generally stick to present tense except for the data taking description. simulation -> simulated. 265: weighted : how? 270: accounted -> accounts 275: toy Monte-Carlo -> peudo-experiments. 280: "official" ? 289: Table.. Figure (also elsewhere) Table 3: S-wave Fig 4: - There are 2 dotted lines but no dashed line. Please change the figure to match the caption. green line -> green dotted - Are the events weighted? In that case you need to correct the y-axis label. - The label 'LHCb' covers part of the error bar on the first data point in the left plot 295: effects -> uncertainties 298: give 1.7 additional... does not really mean anything. 304: remove "with an uncertainty of" Table 4: - BR is undefined (and should be roman) - remove 'item' in 'total item' - Purity -> S-wave fraction (?) 320: why do you suddenly allow for K+pi- combinations? It's Bs->phi(KK)K*bar(K-pi+) you measure. 322: "a significance of 6.06sigma" --> "a significance in excess of 6 sigma" 326 and 330: "result" -> "central value". - if the result agrees within one sigma with the expectation, then why would you say that it is 3 times higher? you clearly don't have the sensitivity to make such a statement. - we presume the central value of the theoretical expectation and Vtd/Vts are actually the same thing, so way say this twice? 334: of -> for 335: "disagrees" is a far too strong statement: the difference (0.37 +/- 0.19) is less than 2 sigma. 337: remove 'very' 340: The statement "This may suggest the presence of phenomena not yet understood" is rather vague: we would call an unaccounted systematic error also a phenomena that is not understood, but probably that is not what you mean.

Refs: use the LHCb refs from the template. [1] and many others: Collaboration -> collaboration [6] and many others: BABAR -> BaBar. BELLE->Belle [15] No title for Ali? [18] and many others: Remove capitals in title. [24] should be replaced by the trigger paper. [30] why CASCADE? It's just the word "cascade" [32] is that the correct title? [33] Remove paper numbers.