I usually ride a 54cm (100mm stem), but I have a bike 56mm with a shorter 80mm stem. It works well. It works even better for gravel as the larger size gives me a longer wheelbase, more stack, and more stability. I think it works well for a gravel bike. Fit is key and is personal, but it is not unusual for people to go up in size for stability, or down in size for agility.
Not sure that I would change a stem more than 30mm though. That will change how fast the steering reacts to your input (you can simulate it to some degree by riding with your hands inboard on your bars – much closer to the stem.