Skip to content

Support high-dimensional BO #499

Open
Open
Support high-dimensional BO#499
Labels
enhancement

Description

@jacktang
jacktang
opened on Jul 15, 2024

Is your feature request related to a problem? Please describe.
Bayesian optimization for high-dimensional constraint problem

Describe the solution you'd like
The solution called Sparse Axis-Aligned Subspace BO (SAASBO) is described in the paper: https://arxiv.org/abs/2103.00349

References or alternative approaches

Are you able and willing to implement this feature yourself and open a pull request?

  • Yes, I would like to try to implement it

Activity

  1. jacktang
    added
    enhancement
    on Jul 15, 2024

  2. t-muser commented on Jul 15, 2024

    @t-muser
    t-muser
    on Jul 15, 2024
    Member

    Hi @jacktang,

    I haven't had the time to read this paper, but considering that this is essentially an acquisition strategy, do you think you could implement this simply by adding a new acquisition function?

  3. jacktang commented on Jul 15, 2024

    @jacktang
    jacktang
    on Jul 15, 2024
    Author

    Hello @Till-M

    Yes, I am going to implement it the way you suggested. And I noticed that you had merge the acquisition function branch to master. Great job! 👍

  4. MandaKausthubh commented on May 30, 2025

    @MandaKausthubh
    MandaKausthubh
    on May 30, 2025

    Hi @jacktang,

    Can we use a Python Library, such as numpyro/pyro, for this, which simplifies and provides a pre-packaged version of the No U-Turn Sampling required in SAASBO?

  5. jacktang commented on Jun 2, 2025

    @jacktang
    jacktang
    on Jun 2, 2025
    Author

    Hello @MandaKausthubh, actually I used numpyro for inner saasgp class. now i am stuck in solving the high dimensional constrained problem, and plan to submit the code in next two or three weeks.

  6. t-muser commented on Jun 2, 2025

    @t-muser
    t-muser
    on Jun 2, 2025
    Member

    Hi everyone!

    thanks for the interest in making this happen. I'm a bit worried about the added complexity of having another GP model just to add SAASBO, I was hoping we could just use the sklearn implementation. Let me think about the best way to move forward.

  7. jacktang commented on Jun 2, 2025

    @jacktang
    jacktang
    on Jun 2, 2025
    Author

    Hello @Till-M, SAASBO may bring some changes : 1) sample based GP (SAASGP) rather than analytical GP (standard GP from sklearn) 2) Sample-based acquisition functions.

  8. t-muser commented on Jun 4, 2025

    @t-muser
    t-muser
    on Jun 4, 2025
    Member

    Hi everyone,

    I did some quick research and there have been two papers in the last two years which discuss whether BO is really performing poorly in high dimensions.

    I'm thinking it might make more sense to first implement some of the results in the second paper, which suggest using a matern kernel, making it anisotropic and also emphasizes an initialization of the lengthscale parameter with $l_i = c \sqrt{d}$ ($c \approx 1$).

  9. jacktang commented on Jun 5, 2025

    @jacktang
    jacktang
    on Jun 5, 2025
    Author

    Interesting! I'd like to spend some time to read the second paper and the code

  10. jacktang commented on Jun 14, 2025

    @jacktang
    jacktang
    on Jun 14, 2025
    Author

    I organized the HDBO implementation which mentioned in paper "Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization" here. Check it if you are interested in it. And try to implement the idea by sklearn GP in next weeks.

  11. t-muser commented on Jun 18, 2025

    @t-muser
    t-muser
    on Jun 18, 2025
    Member

    Hey everyone!

    Thanks for your continued interest in this problem and the will to contribute and make this package better!!

    @jacktang thanks for writing the HDBO code. I'm unfortunately a bit busy right now and can't really have a look at it. Are you planning to replicate the experiments of the paper with it?

    @MandaKausthubh thanks for drafting the PR! I think for now, I don't think the added complexity is worth it, especially given the papers above.

    For now, with regards to High-dim optimization I propose the following: Let's try and make it easier for people to use the results of the "Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization" paper, while seeing how the literature develops. For this we need to add:

    • feature scaling to [0, 1] (which is an assumption made by the paper) -- I've drafted code for this already and can push it soon
    • a simpler way of initializing the kernel anisotropically with their recommended length scale setting -- though maybe this is not needed if we do a good job with the next point:
    • A documentation notebook

    Even if the research in this direction moves towards another approach, these changes are small in overhead and probably useful in any case (the documentation we could just delete).

    What do you think?

  12. jacktang commented on Jun 18, 2025

    @jacktang
    jacktang
    on Jun 18, 2025
    Author

    Hello @Till-M , I had interests to learn the paper, and now only implemented the recommended HDBO by gpytorch and tested it with the high dimensional problems . It's great you've drafted the code and ready to push :). I'd like to test the implementation especially for the constrained HDBO if needed.

  13. t-muser commented on Jun 23, 2025

    @t-muser
    t-muser
    on Jun 23, 2025
    Member

    @jacktang for the feature scaling see this branch of my fork (especially also the diff to master). This does implement a uniform length scale as of now, the anisotropic can be set via set_gp_params.

  14. jacktang commented on Jun 28, 2025

    @jacktang
    jacktang
    on Jun 28, 2025
    Author

    Hello @Till-M , I add Hartmann6 problem for HDBO, the conclusion is current sklearn GP can't solve the optimization problem. I guess feature scaling and length scale are some part of the key to solving the problem, but may be not the all(?)

  15. t-muser commented on Jul 2, 2025

    @t-muser
    t-muser
    on Jul 2, 2025
    Member

    Hey @jacktang,

    great that you worked on the implementation! I had a quick look at your code and I notice you didn't set the kernel to be anisotropic (what they call "ARD" in the paper). You can do that by adding this line before running .maximize

    from sklearn.gaussian_process.kernels import Matern
optimizer.set_gp_params(
    kernel=Matern(nu=2.5, length_scale=np.ones(dim) * np.sqrt(dim)),
)

    NB: This will significantly increase the runtime, and I still didn't get close to the score you mentioned in the markdown cell 🤔

  16. jacktang commented on Jul 29, 2025

    @jacktang
    jacktang
    on Jul 29, 2025
    Author

    Hello @Till-M , I also tried to implement it by using sklearn GP and did not get the great result. I also compare both GP regressors, and the modern GP which using gradient descent with Adam/RMSprop may be more robust?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions