I can't be too clear as I definitely missed some points through MTL but I'll try my best to answer according to my understanding.
The previous emperor (PE) had two sons who were up for the throne: one was the son of his wife (main wife/empress), the future emperor Yun Qing (YQ); the other was the son of his favored concubine/consort, Huiwang. Basically, the emperor didn't like his wife and therefore didn't like her child and he went the opposite direction with the favored concubine. Even though Yun Qing was the crown prince, because PE did not like his mother, and therefore him, he tried to strip YQ of the crown prince title and give it to Huiwang, the favored son. It's the PE's poor treatment and humiliation of his mother, who is his (main) wife, that causes YQ to make sure he never favors any concubines over his own wife and that only his oldest son from his wife is groomed for the throne. His intention is to make it so that his sons will not have to fight each other for the throne when he passes as only one of them will be qualified to inherit; thus, he cultivates the first son but ignores the second which ultimately backfires as the second son's resentment just builds up over time. When the second prince usurps the throne, one of the first thing he does is have his father tortured (conversely, he treats his mother--who admits to having spoiled him to make up for his father's shortcomings--well and other than possibly being under house arrest (it's not stated explicitly but implied...that is, he doesn't force her to remain in her palace but he does have a lot of guards around it and keeping an eye or her/her palace), her life doesn't really become bad or anything. She's not starved and the reason the second prince's wife, (XW) survives his reign is because he won't kill her when she's with his mother....and knowing this, his mother has XW live with her.
The current dynasty (the dynasty of PE and emperor YQ) is headed by the Jiang family.
The ML actually descends from a Jiang princess. A distant grandmother (I can't remember how far removed she was; I want to say it was his great grandmother though) was born a princess of the Jiang family but her family disapproved of her behavior/marriage and crossed her out of the family record, disowning her. So the ML has royal blood in him but is not officially recognized as part of the royal family. His mother (I'm gonna just call her Mrs. L) was a noted beauty in her day and YQ was fond of, but not romantically interested/in love with, her. A good part of the ML's stoicism and likely why he is so willing to indulge the MC's whims from the getgo, before he falls for her, is likely due to how strict and controlling his mother was towards him, not allowing him any semblance of a childhood. The ML is the last living person of his line/family, his mother, father, and older brother having passed away before the story begins. All three had been poisoned (by YQ) with the same poison; Mrs. L passed away first and then about a year or two later, the ML's father and older brother passed away successively around the same time period. The ML didn't know until he was older that their deaths were caused by YQ, who feared that their family was becoming too strong/a threat to his power. He had no proof of them doing anything but had by then become rather paranoid.
One of the reasons the ML is able to survive to adulthood is because he is taken in by the Empress parents, aka, the emperor's in-laws, and raised by them. He is treated well/as if he is one of them by the Empress' parents and is very fond of them, particularly the grandfather. The MC notes that this was extremely beneficial to YQ because having lost his family and then essentially being adopted by the emperor's in-laws meant that the ML was basically indebted towards saw the in-laws as family (since they are his in-laws, the emperor is naturally not worried about the empress' family wanting his throne) and he basically had 24 hour spy access to the ML. I will note that the ML wasn't close to anyone in his biological family--his mother and brother were both noted as being cold/dismissive towards him. However, his parents did love each other deeply and the ML believes even had his father not been poisoned, he wouldn't have lived much longer as he pretty much lost his will to live after Mrs L passed.
Now, for the MC's grandparents...well, the grandfather was targeted twice by both the PE and later, YQ. The MC comes from a long line of accomplished military people and her grandfather was particularly revered (its often mentioned that its a pity the MC's grandfather ended up with a useless son and grandson; many also lament that the MC was born a girl bc she clearly took after her grandfather). As a military commander, the grandpa was normally stationed by the country's borders (I believe the northern border in particular) to fend off against invaders/raiders. The grandfather himself is not blood related to the royal Jiang family (it's the main character's grandmother who has Jiang blood in her; I don't remember if she was a cousin or sister of the PE though) but had very strong military powers--like, his men were noted as being particularly loyal to him and his influence wasn't just contained at the border he was guarding; his abilities/military prowess/person was admired pretty much universally throughout all the military men (this may be due to both his reputation preceding him and the fact that he was genuinely good and caring towards his men and nurtured his fair share of later military leaders as well).
Anyway, as you can imagine, the PE wasn't too excited about so much influence/power over the military being held by someone else outside himself. He either planted or was able to manipulate a trusted officer in the camp to sneakily attack and injure the MC's grandfather. At that point, the grandfather decided it would be best for his family's safety if he resigned from his position and did so. Naturally, MC's grandparents know who is pulling the strings behind the scene in this matter and YQ also learns this incident while he's still fighting for the throne. He's actually very embarrassed/ashamed about his father's actions and feels guilty towards his aunt/MC's grandmother.
Anyway, YQ successfully ascends to the throne and at first, all is well. Gradually, he becomes more like his father in that he becomes paranoid about losing power. The ML's father was a close friend of his but he was also a well-regarded/influential figure in the capital in his own right so the increasingly paranoid YQ decided he had to go. The MC's grandfather, despite having retired and basically spending his days with his grandkids, was still very much so a revered and influential person amongst military personnel; though he retired from his position, he still took the MC to visit military barracks and cared for (wounded) veterans (the MC mentions they have a lot of veterans in their estate and the family's guards are noted by many as being particularly talented fighters). He might not have an official military position but frankly, he doesn't need it because he has the hearts of those in the military and his words/opinions still hold a very heavy weight with the military. YQ also has the MC's grandfather poisoned though it's not stated whether or not it was the same poison as what was used on the ML's family. The MC, who was ~10 at the time, thought he was sick; it's not until the ML reveals how his family was poisoned and asks about her grandfather that she thinks back and realizes her grandfather wasn't ill, but poisoned.
As for the relationship/problems between Mrs L and the MC's grandmother? I'm not sure if the story ever clarifies it. Certainly their personalities would clash (Mrs L was all about being devout and virtuous, or at least projecting that appearance; the MC's grandma is...well, just look at her son and grandkids...her daughter-in-law is also noted to take no bull and she and her daughter in law get along swimmingly...as in, the MC's father wonders more than once whether or not he is actually the grandmother's blood or if his wife is. The grandma apologizes for straddling the daughter-in-law with a husband like her son.
