It is a very difficult question.I went back too soon after 2 weeks which lead to mistakes,fortunately the consequences were not major as my job wasn't major,so that is certainly a consideration.
I have seen it happen quite often on here that people have had to back on sick again when they go back too soon,and others who have been afraid to return but then found the work routine helpful.Again though if yours is anything but routine perhaps that wouldnt work for you.
One thing you might consider is a phased return so you go in for short times,fewer days, or taking back just a few duties, gradually building up to full work over a couple of weeks.Your boss sounds very reasonable so might consider that.To be honest other people who havnt been in your shoes should not be a priority because they cannot comprehend what you are going through so don't let them influence it.If you feel not ready,then you are not ready.