I mean two things. First the study was only n = 20, with 10 subjects + 10 in the control group. This is a very small sample.
Secondly, it's possible the test used to assess memory isn't suitable as mentioned by the authors:
Notably, we did not find a significant association between mTL volume and the performance in visual working memory. N-Back performance, as the only available measure in memory functions in the current project, might, however, not be a suitable task to detect neurocognitive changes related to mTL GM (Jeneson and Squire 2011). In fact, performing N-back task involves majorly frontal and parietal regions (Wang et al. 2019). Moreover, a change in brain structure does not necessarily come with a difference in behavior, as differential brain activity patterns can compensate or adapt to structural changes such that behavioral performance remains unchanged (Barulli and Stern 2013; Rudrauf 2014). Therefore, we strongly encourage a further investigation in functional activities during working memory as well as other cognitive domains under daily caffeine consumption.
I wouldn't stop drinking coffee based on this study.