I received a PhD from the University of Texas
at Austin in geological sciences.
For my PhD, I studied methane hydrates and carbon dioxide
hydrates.
And in particular, we were trying
to understand how to store carbon dioxide as hydrate
in deep marine environments or sub
arctic environments while also producing
methane hydrate for profit.
I finished in 2018.
And after that, I was a data scientist at a small startup
called Slingshot Aerospace.
And I was there for about nine months before joining Novi.
And I guess it was March of 2019.
Does that sound right?
Yeah.
Just before me.
Yeah, exactly.
Like six weeks maybe.
Congratulations on two years.
Yeah, right.
Thank you.
That's what--
in startup years, that's like you've been here for 15.
Oh, yeah.
Yeah, definitely.
How many sprint cycles is that?
Way too many, over 100 at least.
Yeah.
Excellent.
Yeah, well it's-- obviously, you do have a lot of strong
contributions to Novi.
So the reason we're talking today
is because of the forecast uncertainty,
which you've been doing a lot of work on primarily Novi for--
when did you start work on that?
Six months ago?
Nine months ago, something like that?
Let' see.
I think it was over a year ago.
It was in the early part of 2020.
And we were trying to understand how
to do it better than previous methods
that it basically failed.
Yeah.
We were using a lot of trees for estimators,
and that's still the case.
So random or extra trees, et cetera, XGBoost.
And we were trying to bootstrap our way to some uncertainty
profiles.
And that was really cumbersome computationally expensive.
And we stumbled upon this idea confirm what
inference that seems promising.
And it has turned out to be quite fruitful.
So we implemented that for our pre-drill forecasts.
And then found some additional wrinkles we could add and fold
into that to make it really productive for our PDP
forecasts.
Excellent.
So why were you so excited about this new methodology
that you developed?
Why is forecast uncertainty such a difficult problem
with current methods?
Yeah.
Well, for a bootstrap, what that means,
is we're basically building in number models.
And you need to build a lot.
So you could say for a model that has 5,000 wells, maybe
you need to actually still build 1,000 individual models to give
some sense of uncertainty.
And that becomes really expensive,
especially when building one model
takes a couple of minutes.
And if you're also folding into that whole process
type, hyperparameter tuning, feature selection,
suddenly you don't have the compute power
to do that effectively.
With conformal inference, we'd found a special variation of it
called split control inference where we basically just build
two models.
One is about half the data set.
And then the other model is used to evaluate how good we
did on the other half.
And with that, with only two models as opposed to 1,000,
we get most of what we need.
And that's phenomenal.
Awesome.
Now, it sounds like a pretty--
I don't want to go out and train, like,
1,000 different models to get an uncertainty.
That's kind of-- yeah.
So what are the main reasons that you
hear coming from our customers that they're
interested in the uncertainty around our forecasts?
Yeah.
I would say it's the same reason that anyone would ever
want uncertainty around the forecast.
So I would say from everyday life,
you're already incorporating uncertainty
implicitly in a lot of things that you do
and sometimes, explicitly.
So as an anecdote here, I'd say I'm an avid skier.
I recently moved to Montana.
So that I could ski most weekends.
And at my local place, I can be there
in 45 minutes, which means I can wake up early in the morning.
And I can check the weather forecast
and decide if the snow is going to be good,
and the conditions are going to be right.
But that's maybe only a couple of hours in advance.
And I still don't actually know what's going to happen.
The weather forecast might say there's going to be
one to seven inches of snow or absolutely none.
And that matters to me especially if I'm not
trying to ski locally.
I'm trying to ski maybe out of state
and that requires an Airbnb reservation,
that requires driving.
And anticipating what the climatology
is going to look like weeks in advance,
all that carries with it risk that I
want to be able to incorporate in my decision making process.
And the same is true for these operators.
If we're forecasting one quarter in advance,
there's going to be a certain amount of uncertainty
associated with that. And
it's probably going to be small, but it's not zero.
We can't just provide a number to say,
this is exactly what we think is going to happen.
We have to provide a range to make it useful.
And now, as we forecast not one quarter in advance
but maybe a year in advance, two years in advance,
that uncertainty profile should grow.
And as that uncertainty profile grows,
so should the decision making process and the risk
evaluation.
Awesome.
That's a good metaphor.
I'm sure as I see it from Twitter and social media,
it seems like the entire oil field decamps to the slopes
this winter.
So I'm sure that's going to hit home
for a lot of our engineering and executive customers.
Totally.
Yeah.
One thing that we do hear a lot of times
when we bring up this uncertainty that we produce
is a lot of people seem to have this idea
of some kind of a Monte Carlo as their traditional way
of estimating what the uncertainty for a forecast
could be.
So can you give us a little compare and contrast
between our method and a Monte Carlo?
Yeah.
I respect the Monte Carlo method a lot.
And I like it in specific use cases.
One of the backbones of that and requirement, especially
of Markov chain Monte Carlo is that you
have a parametric model.
And what I mean by that is like a linear model is
a parametric model where you say,
I'm going to apply a certain weighting to this feature,
that feature, et cetera.
And then linearly combine them to produce an output.
Any parametric models doing that not necessarily
in a linear way.
So there could be transformations internally
that occur, such as in a neural network where you're feeding
in linear combinations to go through a non-linear function
and then another layer, another layer, et cetera.
For MCMC, you're basically just varying
sets of those internal parameters
and evaluating the space of outcomes
to see which of the parameter sets
is most advantageous for you.
And if you do have a parametric model,
that's a really good idea.
For us, we found a lot of success
with these tree-based models, which are non-parametric,
in the sense, that what you do instead
is you look for corollaries that relate
to your well of interest.
And you say, I've got six wells over here,
six wells over there.
They're very good and similar wells to the one
that I'm trying to predict on.
So I take s average of them.
And because of that, the nature of the way
that we make that prediction, that
means we can't really use MCMC.
And we have to use some other methodologies such as split
control inference or bootstrapping
that allows us to get at the same variables.
And split control inference as I said earlier, is really
are computationally efficient way of doing that.
Yeah.
That sounds pretty powerful.
If my understanding is correct, another big advantage
of our method is that it's actually
measured against actual results on a hold outs
that you can perturb the different parameters,
whether that's in a neural network model
or in a fit to get some idea of what
the distribution is.
But you still have to make some assumptions
about what the distributions of the parameters are, correct?
Yeah you do.
And you don't always.
So I'll modify that question a little bit
to say that you can set it up that way
where you set all of your data into the uncertainty creation
mechanism.
And then you go out into the world,
and you just blindly hope that it works.
Yeah.
You can also set it
up such that you leave some of that data out
and say, I'm going to build my uncertainty quantification
on 80% of my data and then 20% of it
is just going to be out there in the ether.
So then I can evaluate and see how well I did on it.
So that's an evaluation technique using the machine
learning a lot.
And you can apply the same technique and uncertainty
quantification. But
what we do specifically at Novi though
is we ensure that that 80% that we start to build anything on
is sequence in time.
So we use past data or intervals of past data
to then make predictions and uncertainty forecast
on future data.
So then we effectively emulate the process
that any operator would do with our software.
So then we provide a nominal coverage
as we call it of the forecast.
So we're saying, if you want your P90 and your P10,
that's effectively like saying I'm going
to miss out on 20% of that.
And we guarantee it with our simulation
of past trained forward-looking future predictions.
Yeah.
That's a really good distinction that sometimes we
lose sight of it in oil and gas industry
and everything that we're making forecast for has a time basis.
So could you draw a little distinction
between what is the error look like on the hold
outset for my entire data set versus for a single well?
Yeah.
I think it totally depends on the data
set and the variables of interest within that data set.
But effectively, when you're trying
to aggregate on a specific IP day
or a number of forward forecast steps,
you often have to reduce that to some statistic.
So you might say, I'm going to look at the median area of all
of the wells of my interests at this IP day
or at this forecast step.
And it gives you some percentage of how far
you might be off for a number of barrels, et cetera.
And you might even then expand that to say,
I don't want to just look at P50,
I want to look at P25, P75.
But as you aggregate that, it's really only telling you
how good Model A is against Model B.
And you would hope that Model A also on the individual level
if it's better than Model B,gives a tighter uncertainty
band on every individual forecast.
But the error that it has that it holds
does not necessarily relate to the individual well
where the individual well is dependent on each specific
feature, and how that feature is part
of the collection and distribution of all
the rest of the features.
So for instance, if you're going to make a forecast
and you're going to introduce some proppant that
is at the fringes of the profile of the property
that you've seen in your training,
you should have a really big uncertainty profile
even if your model is incredibly accurate.
Because your model saying it's incredibly accurate
meant that you were evaluating on the distribution
of proppants that you'd already see.
Yeah. So
in some ways, what you're saying
is that the model may be extremely
confident in its forecasts for a well, which
is very similar to the other training data that it's seen.
Whereas, if you send it something totally new,
it might have huge uncertainty, just
like if your meteorologist's forecasting your snow
conditions up in your local Montana mountain,
if you ask the same meteorologist to give you
snow conditions down in New Zealand,
then that person might have a lot higher uncertainty
about the forecasts that they're making
for how many inches of snow you're getting.
Yeah, absolutely.
And sometimes we call these exogenous features
or from an operator's point of view,
it's interventional choices that have been made.
The uncertainty should reflect the degree of difficulty
or the degree of variation against the norm.
And we do bake that in.
We ensure that the predicted forecast uncertainty
we have accounts for every individual feature
that you're putting into the model in addition
to the past production of that profile.
So you could end up with high uncertainty because
of the forecast or the actual previous production is erratic,
or because you're introducing a feature that
is really far from the behavior seen by the model.
Interesting.
So what are some of the applications
that you're hearing about from our customers
for these type of uncertainty bands or confident bands?
I think the primary use case is evaluating risk.
So if you're going to go out and you're
going to drill X number of new wells,
and you're looking in geographical fringe area,
you want to know how good is the model that you thought
was good but in an area that you don't actually
have much experience.
So you're extrapolating beyond the bounds
of the data or right at the edge of the bounds of the data.
Or what you might be doing is evaluating
your entire inventory of wells, and you
want to go X number of quarters into the future and project
revenue.
The revenue can't just be a finite number.
It needs to be a range of possible numbers given
everything that you know. So
they're using our uncertainty forecast
to determine what the bounds on future revenue might be.
Super interesting.
So is this stuff already in our products right now?
It absolutely is.
It is a standard feature in our products.
And it can be used for pre-drill wells or PDP wells.
And it's configurable to the point--
you may have to check on this.
But I think you can apply to different sets of confidence
interval brackets.
I think that's our limit right now.
So if you were interested in the upper and lower 10%,
you could ask for that.
Or the upper and lower 25%, you can also simultaneously
ask for that.
So somebody might want to get some idea about the tails
of the distribution of any given forecast
is for maybe understanding what their risk is.
But then maybe a better idea of what
the middle of the distribution looks
like for let's say, per well economics or offtake planning
or something like that.
Yeah.
Yeah. And
our point of view is we provide as much information as
possible so that the person asking for the forecast
can then determine what application that information
is useful for, how they may want to use it.
And then as you stated, to be able to use different parts
of it for different use cases.