Open access peer-reviewed chapter - ONLINE FIRST

# Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer

By Aishat A. Motolani, Mengyao Sun, Matthew Martin, Steven Sun and Tao Lu

Submitted: August 14th 2019Reviewed: May 14th 2020Published: July 24th 2020

DOI: 10.5772/intechopen.92830

## Abstract

Cancer is the second leading cause of mortality in the United States. There are several therapeutic regimens employed to mitigate the mortality rate of cancer. This includes the use of chemotherapy, radiation, immunotherapy, and precision medicine/targeted therapy. Targeted therapy involves the use of drugs that target a specific pathway or biomolecule compromised in cancer for cancer treatment. Aberrant expression of epigenetic enzymes has been well documented for their contribution in driving tumorigenesis and other cancer hallmarks. Hence, there is an urgent need for novel drug discovery and development in epigenetics to help combat various cancer morbidities. Herein, we review the roles and consequences of dysregulated function of several epigenetic enzymes, with a focus on histone methyltransferases (HMTs). Additionally, we discussed the current efforts made in the development of small molecule inhibitors for a few representative HMTs implicated in different cancers. Furthermore, the common screening assays used in discovering potent small molecule inhibitors were also detailed in this chapter. Overall, this book chapter highlights the significance of targeting HMTs in different cancers and the clinical application potentials/limitations faced by the developed or emerging small molecule inhibitors of HMTs for the purpose of cancer therapy.

### Keywords

• cancer
• drug discovery
• epigenetics
• histone methyltransferases
• small molecule inhibitors

## 1. Introduction

DNA mainly undergoes methylation, and this occurs through the action of a family of DNA methyltransferases (DNMT1, DNMT2, DNMT3a, and DNMT3b). DNMTs covalently modify DNA by catalyzing the transfer of methyl group from S-adenosylmethionine (SAM) to the C5 position on a cytosine ring. DNA methylation mostly occurs in CpG islands, a region of the DNA rich in cytosine and guanine base repeats [6]. This modification to the DNA functions to repress transcription when it occurs in gene promoters and regulates splicing when it occurs in gene bodies [7]. DNA methylation is a reversible mechanism, which can be either passive through reduced DNMT1 activity during DNA replication or active through activity of its erasers, DNA demethylases. For instance, the ten-eleven translocation proteins (TET 1/2/3) are human demethylases that catalyze the oxidation of 5-methyl-cytosine to 5-hydroxymethylcytosine [8]. Following DNA methylation, often readers such as the family of methyl binding domain proteins recognize the methylation marks to drive transcriptional repression [9].

RNA is also methylated on C5 position of cytosine (m5C) and N6 position of adenosine ring(m6A) by family of RNA methyltransferases such as Dnmt2, NOP2/Sun, Mettl3, and Mettl14. RNA methylation can be reversed by these RNA demethylases: fat mass and obesity associated protein (Fto) and α-ketoglutarate-dependent dioxygenase alkB homolog 5 (AlkBH5) [10]. Methylation modification on RNA is interpreted by readers such as the YTH domain family, and they mediate RNA splicing, export, stability, maturation, decay, secondary structure switch, and translation [11]. There have also been reports of RNA acetylation by NAT10 acetyltransferase, which functions to promote mRNA stability and efficiency in translation [12].

Moving up the central dogma, lysine and arginine residues on histone proteins are mostly subjected to various post-translational modifications by their respective epigenetic enzymes to either activate or repress transcription. Although the focus of this chapter is histone methylation, histone acetylation will be briefly discussed. Histone methyltransferases (HMTs) and histone acetylases (HATs) are the writers of histone methylation and acetylation, respectively. HMTs can be further subdivided into lysine methyltransferase (KMT) and protein arginine methyltransferase (PRMT) [13]. The families and functions of HMTs will be further explored in this chapter. On the other hand, several HATs have been discovered, with the major ones being the GNATs (Gcn5-N-acyltransferases), the MYST families, and p300/CBP [5]. These enzymes catalyze the transfer of acetyl group from acetyl Co A to the side chain amino group on histone lysine residues, inducing transcriptionally active chromatin [13]. Histone deacetylases and histone demethylases are involved in reversing the histone modifications discussed above. The families of lysine-specific histone demethylase 1 (LSD1) and Jumonji histone demethylases (JMJD) mediate the removal of methyl groups from histone through respective mechanisms [13, 14]. Readers of histone methylation include protein containing the MBT, PHD, chromo, Tudor, double/tandem Tudor, Ankyrin Repeats, zf-CW, WD40, and PWWP domains [15].

Given this array of epigenetic enzymes and their broad spectrum of function in regulating several gene expression in humans, the roles of epigenetic enzymes have been implicated in tumorigenesis. The epigenome of cancerous cells has widespread changes in DNA methylation and histone modification patterns [16]. For instance, hypermethylation of CpG islands in the promoter of chromodomain helicase DNA-binding protein 5 (CHD5), a chromatin remodeler, was observed in colon, breast, hepatocarcinoma, cervix, and glioma cell lines [17]. This results in the downregulation of CHD5, which plays a tumor suppressive role in cells. Similarly, hypomethylation of DNA at promoters of oncogenes such as insulin-like growth factor 2 (IGF2) has been observed in breast and colon cancers. The differential methylation patterns on promoters of tumor suppressors and oncogenes mediated by increased/reduced activity of DNMTs and TETs enzymes have been used as biomarkers to predict the predisposition of individuals to cancer [18]. Also, aberrant expression of histone acetyl transferases (HATs) and histone deacetylases (HDACs) has been linked to tumor development. Studies have shown that that p300/CBP acts as a coactivator with c-Myb to activate the transformation of fusion oncoproteins in myeloid leukemia [19]. Increased expression of HDACs was reported in gastric, prostate, colon, and breast carcinomas, and this results in repression of tumor suppressor genes like cyclin-dependent kinase inhibitor, see p21 [20]. Of all the histone modifications, histone methylation dysregulation is mostly attributed to poor prognosis in several cancers, which we will further elaborate in detail in Section II “Histone Methyltransferases in Cancer.”

As these epigenetic enzymes’ activity has been altered in cancer and contributes to the genomic instability in cancer cells, it is crucial to develop targeted therapeutic treatments to restore their normal function. Aside from surgery, some common treatment options for cancer patients can be broadly categorized as thus: chemotherapy, immunotherapy, radiotherapy, and precision medicine/targeted therapy [21]. These classes of treatments are not mutually exclusive and can be used in concert for treating cancer patients. Among these different therapeutic approaches, targeted therapy is the future for cancer treatment. Targeted therapy involves the use of drugs that target a specific biological molecule/pathway or drug treatment that requires genome profiling of an individual before it can be administered [22]. For optimal development of drugs for targeted therapies, it is important to identify a well-defined biological target whose activity contributes to one to several hallmarks of cancer including propagating growth signals, evading immunosurveillance, cell death resistance, activating metastatic programs, suppressing antigrowth signals, inducing angiogenesis, and enabling immortal replication of cells [23, 24]. For example, cancer patients whose tumors are driven by high activity of epidermal growth factor receptor (EGFR) signaling can be treated with specific monoclonal antibodies or small molecule kinase inhibitors antagonizing the aberrant signaling, and thereby reducing tumor proliferation [25]. Similarly, targeting an epigenetic enzyme, PRMT5, which is highly expressed in gastrointestinal cancers, with small molecule inhibitor, PR5-LL-CM01, was shown to slow down cancer cell growth and invasion in vitro [26]. The limitations of targeted therapies include cancer cell resistance to drug treatment by activating a parallel pathway or sometimes targets can undergo mutation, making drug accessibility to target difficult [27].

Given the myriad of biological targets that have been discovered to mediate cancer progression, there has been increasing interest in the development of small molecule inhibitors capable of decreasing the activity of those targets. Small molecules are intracellular targeting compounds with low molecular weight of less than 900Da. They can modulate their target activity as an agonist or antagonist [23]. The growing interest in the use of small molecules for drug development is not only due to their small size, which enables easy permeability into the cell, but also their desirable pharmacokinetics, pharmacodynamics, longer shelf life, and easy synthesis [28]. Several small molecule modulators have been developed into drugs to treat various types of cancers. The range of small molecule inhibitors developed to enable tumor regression can be broadly categorized into small molecule kinase inhibitors, proteasome inhibitors, metalloproteinases and heat shock protein inhibitors, and apoptosis targeting inhibitors [29]. The most common small molecule inhibitors, kinase inhibitors, have been used to inhibit the several protein kinases whose activity is dysregulated in cancers. The first tyrosine kinase inhibitor drug, Imatinib, is a small molecule ATP analog that competitively inhibits Bcr-Abl fusion protein kinase activity in chronic myeloid leukemia patients [30]. Similarly, a number of small molecule inhibitors targeting epigenetic enzymes implicated in tumorigenesis are in development or have been FDA approved for cancer treatment. For instance, drugs like belinostat and romidepsin are HDAC inhibitors that are FDA approved for the treatment of lymphoma [31]. After the first clinical trial in 2014, tazemetostat, an EZH2 small molecule inhibitor, moved to phase 2 clinical trial and was fast tracked by FDA in 2017 for the treatment of follicular lymphoma [32]. The use of tazemetostat for treatment of epithelioid sarcoma in adults 16 years and above was also granted accelerated approval by the FDA. These examples, among many others, show the potentials of epigenetic modifiers as a druggable target for cancer treatment. More of these small molecule inhibitors for histone methyltransferases will be explored later on in this chapter. However, challenges in the clinical application of certain small molecule inhibitors as drugs remain due to their off-target effects or development of resistance by cancer cells [29].

## 2. Histone methyltransferases in cancer

As we mentioned above, epigenetic enzymes, including histone methyltransferases, are novel targets for cancer therapy. In this section, we will review the role of histone methyltransferases in cancer.

### 2.1 Lysine methyltransferases

Lysine methylation of histones was first characterized in the 1960s [33, 34] and was initially described as an “irreversible” post-translation modification. This dogma was challenged by the discovery of histone demethylases by Shi et al. indicating a dynamic function of methylation allowing the addition and removal of methyl groups [35]. The proteins responsible for the addition of methyl groups to histones are known as lysine methyltransferases (KMTs). KMTs are broadly defined as SET [Su(var)3-9, Enhancer of Zeste, Trithorax] domain-containing proteins and non-SET domain-containing proteins [36]. The only non-SET domain-containing KMTs identified so far are DOT1 (disrupter of telomeric silencing 1) family members, which methylate K79 of histone H3 and which also do not share structural similarities with SET-containing proteins [37, 38, 39]. KMTs act by catalyzing the transfer of 1~3 methyl groups to lysine residues of histone and non-histone proteins through the addition of the cofactor S-adenosyl methionine (SAM), which acts as a methyl donor group [40]. In the case of SET domain-containing proteins, SAM interacts and orients with a lysine residue of the substrate histone tail within the catalytic pocket of the SET domain. Then, a tyrosine residue acts to deprotonate the ε-amino group of the lysine residue, which allows the lysine chain to nucleophilically attack the methyl group of the SAM molecule, transferring the methyl group to the lysine side chain [41]. In the case of non-SET domain-containing KMTs, the enzyme DOT1 acts to methylate a lysine residue in the histone core [40]. As we described above, histone methylation is a critical epigenetic modification of chromatin that can impact genomic stability, alter expression of different genes, determine cell lineage, alter DNA methylation, as well as control cell mitosis [42].

SET-containing proteins have been characterized in greater detail than non-SET-containing proteins. SET domain proteins are characterized as seven families of the superfamily of KMTs: SUV39, SET1, SET2, EZ, RIZ, SMYD, and SUV4-20 [36]. There are also several SET domain proteins that do not fall into these groups including SET7/9 and SET8 [36]. The SUV39 family has been characterized the most out of these groups of KMTs. Specifically, Schizosaccharomyces pombe Cryptic loci regulator 4 (CLR4), human SUV39H1, and murine SUV39H2 were among the first identified SET domain protein lysine methyltransferases characterized when their sequence homology between their SET domains was discovered. These proteins methylate lysine 9 of histone H3 (H3K9) [43]. SET1 and SET2 complexes are involved in the RNA polymerase II holoenzyme [44, 45]. TSET1 acts to tri-methylate H3K4 and is associated with early gene transcription as opposed to SET2-mediated methylation of H3K36, which is associated with later transcriptional elongation of downstream genes. The mammalian nuclear-receptor-binding SET domain-containing protein (NSD1, a member of the SET2 family) has an important function in methylation of H3K36 and H4K20 in development [46]. Lu et al. in our research group also reported that NSD1 could methylate non-histone protein, NF-κB, at K218/221 of its p65 subunit, leading to the activation of NF-κB and its downstream target gene expression [47, 48, 49, 50]. Another example is SETDB1, a H3K9 methyltransferase, which is amplified in primary tumors of lung cancer patients and contributes to the invasive phenotype of tumor cells [51]. Additionally, SETDB1 methylates H3K9 in euchromatin, and is also required for development [52]. Human SET7/9 acts to mono-methylate H3K4 [53]. Furthermore, SET domain enzymes’ functions are not specifically tied to histone methylation. Human SET7/9 methylates K189 on transcription factor TAF10, which increases RNA polymerase II affinity and transcriptional activity, thereby expression of TAF10-dependent genes [54]. SET7/9 is also involved in p53 methylation, which increases its stability [55]. These examples described are only a few of the many SET-containing proteins of which over 50 different proteins have been discovered [56]. While SET domain proteins are typically referred to as histone lysine methyltransferases, it may be more accurate to refer to them as protein lysine methyltransferases due to their identification of non-histone targets for these KMTs.

These examples of SET and non-SET domain-containing proteins exhibit the importance of KMTs in the regulation of histone and non-histone proteins. Therefore, it is not surprising that dysregulation of KMT function can result in dysregulation of cellular functions. Specifically several KMTs have been associated with tumorigenesis of several different cancers.

H3K9 methyltransferases SETDB1 and G9a are both known to have roles in gene silencing and embryo development [57]. G9a regulates cancer metabolism in several types of cancer [58]. Overexpression of G9a is associated with worse prognosis in patients with prostate cancer. Intriguingly, G9a knockdown in breast and lung cancer has been shown to promote E-cadherin expression and thus tumor metastasis [59, 60]. Moreover, G9a’s higher expression predicts higher mortality of ovarian cancer patients [61] and is reported to be associated with cell growth and proliferation in neuroblastoma [57]. Furthermore, G9a has been shown to have a critical role in the development of pancreatic carcinoma and acute myeloid squamous cell carcinoma. G9a-dependent repression of genes is also associated with development of leukemia as well as squamous cell carcinomas [57]. SETDB1 is another H3K9 methyltransferase reported to play a role in numerous types of human cancer. SETDB1 is involved in regulation of several cellular processes, including apoptosis, DNA damage repair, and regulation of transcription factors [62]. SETB1 is associated with oncogenic activity and is upregulated in several cancers including lung cancers, gliomas, and prostate cancer [63, 64, 65, 66]. For instance, in lung cancer, overexpression of SETDB1 promotes invasiveness and knockdown reduced lung cancer cell growth. In gliomas, cell proliferation was reduced by suppression of SETDB1. In prostate cancer, downregulation of SETDB1 led to reduced cellular proliferation, migratory ability, and invasive ability.

EZH2 (Enhancer of zeste homolog 2), a H3K27 methyltransferase, plays an important role in transcriptional repression [36]. EZH2 plays a critical regulatory role in as many as 46 types of human cancer [57, 67]. Typically, EZH2 is overexpressed in cancers, and its high expression has been linked to worse patient survival. For instance, EZH2 downregulation in breast cancer can block cell growth and survival [68]. Moreover, EZH2 knockdown inhibits invasive ability and cellular proliferation in prostate cancer [69]. Additionally, EZH2 has also been shown to have roles in gliomas and renal cell carcinoma, wherein decreased EZH2 expression reduces cellular proliferative ability and promotes apoptosis [70].

Another example is SMYD3 (SET and MYND domain-containing 3). It is a KMT that methylates H3K4 and is extremely important for initiation of transcription. High SMYD3 activity is indicative of an epigenetic signature of active enhancers [71]. SMYD3 knockdown in colorectal cancer can impair cellular proliferation [72]. In breast cancer, knockdown of SMYD3 also inhibits cell growth, and overexpression can promote carcinogenesis by regulation of the Wnt signaling pathways [73]. Additionally, reduction of SMYD3 expression in prostate cancer inhibits cellular proliferation, migration, and colony formation [74].

H3K36 methylation is critical for transcriptional elongation, and H3K36 methyltransferases have been identified to play important regulatory roles in several types of cancer. NSD1, a H3K36 KMT, is involved in prostate cancer androgen receptor transactivation, which results in prostate tumorigenesis [75]. Overexpression of NSD1 in neuroblastoma reduces cellular growth ability and colony formation [76]. NSD1 has also been reported to play a role in myelomas and lung cancers [57]. As we described above, our lab also found that NSD1 activates NF-κB to induce its target gene expression. The function of these genes is frequently involved in oncogenic phenotype, or cytokine, chemokine secretion [47]. Another example is H4K20 methylation, which is reported to be important for gene transcription. For example, overexpression of methyltransferases of H4K20 SUV420H1 and SUV420H2 is associated with decreased cell invasiveness in breast cancer, and knockdown increases epithelial to mesenchymal transition [77].

DOT1, the only known H3K79 methyltransferase and non-SET containing KMT, also plays an important role in cancer development. DOT1 is involved in cell survival and colony formation in several forms of leukemia [78]. The activity of human DOT1L (hDOT1 like) methyltransferase is compromised in mixed lineage leukemia (MLL) and is required for maintaining proliferative state of transformed cells [79]. Also, downregulation of DOT1 in lung cancer can cause cell cycle arrest and reduce cellular proliferation [80].

Together, these examples show several of the pathways regulated by KMTs and also highlight the critical importance of tight regulation of these pathways wherein dysregulation of KMTs can result in promotion of tumorigenesis.

### 2.2 Protein arginine methyltransferases

In contrast to KTMs, there is another important family of protein methyltransferases, namely, protein arginine methyltransferases (PRMTs). It is a family of nine members (PRMT1-9) and specifically catalyzes the methyl transfer from SAM to the guanidine nitrogen (ω-NG) of the arginine residues of protein substrates [81]. After the donation of methyl groups, SAM forms S-adenosyl-L-homocysteine (AdoHcy, SAH) and methylarginine is produced [81]. There are three forms of methylarginine recognized in mammalian cells: ω-NG-monomethylarginine (MMA), ω-NG, NG-asymmetric dimethylarginine (ADMA), and ω-NG, N’G-symmetric dimethylarginine (SDMA) [82]. The family of PRMTs is categorized into three major types: type I, II, and III. Type I and II PRMTs first catalyze the formation of MMA, and then type I PRMTs (PRMT1, 2, 3, 4, 6, and 8) further catalyze the formation of ADMA, while type II (PRMT5 and 9) would instead catalyze the production of SDMA. For type III PRMTs (PRMT7), they only catalyze the formation of MMA [83]. These three types of PRMTs methylated similar substrates, such as histone, but they also can catalyze different non-histone substrate proteins. Additionally, although the majority of PRMTs methylate glycine-arginine-rich (GAR) motifs in their substrates [84, 85], some of them display a preference of methylation on the proline-glycine-methionine rich (PGM) motifs in substrate proteins such as PRMT4 [86, 87], while PRMT5 is characterized with the methylation of both types of motifs in proteins [86, 88].

PRMTs are widely expressed in mammalian cells and regulate primary cellular processes, including cell proliferation and differentiation. The abnormal expression of some types of PRMTs such as PRMT1, 4, 5 and 6 frequently leads to tumorigenesis and malignancy.

PRMT1 was the first protein arginine transferase recognized in mammals and assumes the large fraction of arginine transferases activity in mammalian cells [89, 90]. It has been reported that PRMT1 is related to various kinds of cancer. Le Romancer et al. suggested that PRMT1 governed the interaction of estrogen receptor α (ERα) with steroid receptor coactivator proteins (Src), the p85 subunit of phosphatidylinositide 3-kinases (PI3K) and focal adhesion kinase (FAK) [91]. PRMT1-mediated ERα methylation is integral for the activation of the Src-PI3K-FAK signaling pathway [91]. In their subsequent report, the authors further demonstrated arginine methylation of ERα by PRMT1 might remarkably activate protein kinase B (PKB, also known as AKT) [92]. Another example is methylation of Axin by PRMT1. Cha et al. showed that arginine methylation of Axin by PRMT1 may activate the WNT pathway by destabilizing Axin and promote tumorigenesis [93, 94]. Importantly, it is reported that methylation of meiotic recombination 11 (MRE11, also known as MRE11A) and p53 binding protein 1 (53BP1) by PRMT1 can block the DNA repair pathway, contributing to cancer progression [95, 96]. MRE11 combines with DNA repair protein RAD50 and Nijmegen breakage syndrome 1 (NBS1) to form MRE11-RAD50-NBS1 complex (MRN complex). The mammalian MRN complex plays significant role in repairing DNA double-strand breaks (DSBs). Yu et al. reported that the deficiency of arginine methylation of MRE11 in its GAR motif resulted in exonuclease and DNA-binding defects and finally failing to repair DNA damage [96]. Interestingly, Boisvert et al. found that 53BP1 could not relocalize to DNA damage sites and γ-H2AX formation was decreased in fibroblasts treated with methylase inhibitors [95, 97]. Moreover, Mitchell et al. discovered that depletion of PRMT1 affected the length and stability of telomere [98]. Since dysfunction of both DNA repair pathway and telomere maintenance is known to be the cause of cancer, deregulation of PRMT1 may lead to tumorigenesis by these pathways.

In addition to PRMT1, another PRMT member, PRMT4 (also known as CARM1), is reported to be tightly associated with estrogen-mediated oncogenesis of breast cancer through the upregulation of transcription factor E2F1 expression [99]. Moreover, CARM1 is suggested to be overexpressed in human colon cancer and exert a crucial role in Wnt signaling through mediating the action of β-catenin on Wnt target genes as a transcriptional coactivator [100]. Moreover, c-fos is a proto-oncogene and overexpressed in a set of cancers. Some groups reveal that PRMT4 regulates transcriptional activation of c-fos, and that matrix metalloproteinases (MMPs), c-fos target genes, are significantly downregulated in CARM1-deficient cells [101]. Therefore, arginine methylation by PRMT4 is related to multiple oncogenic signaling pathways.

An important PRMT member that plays a critical role in cancer is PRMT5. As a primary type II PRMT, PRMT5 functions in the presence of other binding partners, such as MEP50. Hou et al. discovered that E-cadherin expression was remarkably repressed by SNAIL and PRMT5 recruited by bridge molecule AJUBA, which was favorable for tumor metastasis [102]. It was also reported that p53 can be methylated on multiple arginine sites by PRMT5 in response to DNA damage [103]. Scoumanne et al. proposed that PRMT5 inhibition may promote cancer cells to progress toward apoptosis under chemotherapy/radiotherapy [104]. Moreover, Cho et al. found methylation of E2F1 by PRMT5 weakened its ability to promote apoptosis and repress proliferation, indicating PRMT5 overexpression may enhance cancer cells’ growth and survival [105]. It is well known that continuous NF-κB activation exists in most cancers. Our group uncovered that PRMT5 dimethylated the p65 subunit of NF-κB at arginine 30 (R30) to activate NF-κB pathway [106], an important transcription factor that is involved in the progression of many cancers. Also, PRMT5 is reported to promote its own overexpression in several cancers through a feedback loop involving NF-κB signaling [107].

Besides the PRMTs we discussed above, PRMT6 is widely taken as a transcriptional repressor. Neault et al. reported that embryonic fibroblast cells from the PRMT6 knockout mouse were subjected to a premature senescence, while the cellular senescence can be rescued in PRMT6 and p53 double knockout mouse embryonic fibroblast (MEF) cells [108], affirming growth suppression effect of excess p53 due to PRMT6 deficiency. Thus, PRMT6 actively suppresses p53 cascade to promote tumorigenesis in cancer.

Taken together, many members in the PRMT family have shown essential role in cancer development and progression. Thus, it is unsurprising that these PRMTs have become the rising targets in cancer therapeutics in recent years.

## 3. Discovery of small molecule inhibitors for histone methyltransferases in cancer treatment

The timeline of drug development from conception of the idea to a feasible drug available in the market for treatment of diseases takes between 12 and 15 years and can cost up to $1billion [109]. Drug discovery process begins with identifying a druggable biological target that contributes to a disease progression. These targets can be identified through text mining from online databases, microarray data mining using bioinformatic tools, proteomic data mining from proteomic databases, and chemogenomic data mining, which involves simultaneous exploration of multiple cell phenotypes by screening small molecules from chemical libraries to a number of biological targets [110]. Then, promising or known targets can be validated in vitro and in vivo to confirm that their activity influences phenotype associated with a disease. This step is followed by screening or high throughput screening (HTS), which describes the process of sifting through compound libraries for molecules with high affinity for a target of interest [111]. The two approaches of developing assays for compound library screening are the biochemical target-based approach and the cell-based approach. Biochemical target-based approach is often employed as the primary screening assay in epigenetic compound screen because it allows the direct monitoring of ability of a target activity as opposed to cell-based assays wherein changes in cell phenotypes are measured [112]. Listed below are some of screening assays used in the preclinical development of drugs for epigenetic enzymes. ### 3.2 AlphaLISA screen (amplified luminescent proximity homogeneous assay-linked assay) AlphaLISA is a high throughput screening approach used to analyze and measure post-translational modifications, protein-protein interactions, and concentrations of analytes. The robust, highly sensitive, reproducible, miniaturized, scalable, and automated nature of AlphaLISA assays earned them their widespread application in research and drug discovery. The principle behind AlphaLISA technology is based on the mechanism of another methodology, Luminescent oxygen channeling immunoassay (LOCI). LOCI involves chemiluminescent reaction of a singlet oxygen transfer and was developed in 1994 by Ullman et al. to quantify latex particle binding [113]. Similarly, AlphaLISA assays employ biotinylated antibody bound to streptavidin donor beads and an acceptor bead bound to a second antibody. These antibodies are specific to different epitope on a protein (could also be product of a bimolecular interaction or a modified protein). The binding of these antibodies to the proteins brings the donor and the acceptor bead into proximity of at most 200nm. Upon excitation of donor beads at 680nm, a singlet oxygen is excited from the donor bead and this triggers singlet oxygen reaction with the chemical dyes (thioxene and europium) on the acceptor bead, which results in chemiluminescent emission at 615nm [114, 115]. Multiplate readers equipped with AlphaLISA screen detections can be used to record signals. A systematic method used to validate the quality of HTS assay output like AlphaLISA is called the Z-factor. The Z-factor is calculated by accounting for the positive and negative controls’ mean signal-to-mean background ratio. Hence, in the design of AlphaLISA assay, to ensure quality control, wells containing maximum signal and no signal solution mix must be included [116]. This assay is widely optimized to screen for small molecule modulators for different epigenetic enzymes. For example, in our lab, Prabhu et al. used an optimized AlphaLISA screen protocol to identify a lead compound capable of targeting PRMT5, and subsequently, the compound was shown to be more potent in reducing pancreatic and colon cancer cells’ proliferation compared to another commercially available compound EPZ015666 [26]. Not only is this assay used to screen for compounds for epigenetic targeted therapy, but also they uncover new roles of different epigenetic enzymes in cancer. Consequently, together with other assays, it was uncovered using AlphaLISA HTS assay that an inhibitor of G9a lysine methyltransferase, A-366, limits cell growth and differentiation in leukemic cells [117]. Potent small molecule inhibitors for EZH2, a methyltransferase known to silence tumor suppressor genes, are also being identified using this particular HTS assay [118]. AlphaLISA kits specific for histone methyltransferase modifications, among many other epigenetic enzymes, are commercially available for research and drug development purposes [119]. ### 3.3 FRET (Förster/fluorescent resonance energy transfer) assay Discovered in 1946 by Theodor Förster, FRET is cell-based assay that enables real-time observance of molecular interactions within cells [120]. This phenomenon depends on the proximal interaction (1-10nm) of a donor fluorophore and an acceptor fluorophore, whereby upon excitation, the donor fluorophore transfers energy to the acceptor, increasing its emission wavelength [121] (Figure 1). The measure of FRET is taken as the ratio of the intensities of the donor/acceptor fluorophore. Although highly sensitive in distance, FRET assay does not permit the level of sensitivity and high throughput as AlphaLISA [122]. A type of FRET, FLIM-FRET (fluorescence lifetime imaging FRET) was used to conduct epigenetic biomarker screening in ER-positive breast cancer cell line and patients. The utilization of FRET in this study was based on the presence of certain histone modifications around ERα in the nucleus. Consequently, the assay revealed H3K27ac and H4K12ac interaction with ERα, making HATs a potential therapeutic target in compound screening [123]. Another study optimized TR-FRET (time-resolved FRET) for high throughput screening, following the treatment with HDAC inhibitors, to detect the methylation levels of histone 3(H3) in U-2 OS cells using terbium-tagged antibodies specific to a particular H3 modification and green fluorescent protein (GFP)-tagged H3 [124]. ### 3.4 In silico screen In silico screening or virtual screening is a common method used in drug discovery as a pre-filtering method for identifying promising compounds that can be used for experimental studies. Drug development using this method of screening is estimated to save approximately$130 million and 0.8 years per drug [125]. The approach to virtual screening can be broadly divided into structure-based methods and ligand-based methods. Structure-based approach encompasses docking candidate molecules against available 3D structure of the target protein. When there is no crystal structure of the target protein, ligand-based approach is more useful because it relies on the screening of bioactive ligands of a similar chemical structure [126]. This approach is similar to pharmacophore-directed homology modeling: a process that involves superposing known active ligands for structurally similar targets and then extracting matching chemical properties of the ligand that are required for their bioactivity. Pharmacophore from different bioactive molecules can be generated using commercially available software such as HipHop, PHASE, DISCO, HypoGen, among others [127].

In computer-aided design of epigenetic drugs, there are a plethora of databases like ZINC containing over 35 million compounds available for screening. Other databases like SPECS, Chembridge, and Enamine have been used to identify inhibitors for most subsets of histone methyltransferases [128]. Molecular dynamics simulation also aids in drug discovery as they are employed to understand the conformational changes in the different domains of a target protein [129]. For instance, a study developed analogs of eosin, a template molecule known for having anti-methyltransferase activity, using pharmacophore methods. These molecules were docked on to PRMT1, SET7, and CARM1, and the AutoDock analysis revealed that compounds that target SAM substrate-binding site were more active in PRMT1 and CARM1 while those that target lysine and co-factor binding site were more promising in SET7 [126].

## 4. Current small molecule inhibitors of histone methyltransferases

Development of small molecule inhibitors for histone methyltransferases has garnered remarkable attention over the past years due to their combined efficacy and potency in various cancer treatments. In this section, we will discuss small molecule inhibitors of a few HMTs that have either shown promising results in preclinical development, clinical trial stage, or that have been FDA approved.

### 4.1 Small molecule inhibitors of EZH2

As aforementioned, EZH2 is a lysine methyltransferase that is overexpressed and found to contribute to many cancer progressions including but not limited to breast, prostate, colon, ovarian, liver, bladder, lymphoma, skin, and lung cancer. The overabundance of EZH2 causes hypersilencing of genes that restrain proliferation and promotes differentiation [67]. As a result, several studies have been conducted to understand the mechanism of action and structure of the enzyme so that appropriate drugs can be developed to inhibit its aberrant activity. For example, FDA has approved the use of tazemetostat (EPZ6438) (Table 1), an EZH2 small molecule inhibitor, for the treatment of epithelioid sarcoma (not qualified for resection) in 16 years and above patients. Tazemetostat has an inhibition constant (Ki) of 2.5nM and works by competitively inhibiting SAM binding site on EZH2 [32]. Another small molecule inhibitor, CPI-1205 (Table 1), completed phase 1 clinical trial for B-cell lymphoma and solid advanced tumor and is in phase 1b/2 clinical trial for metastatic castration-resistant prostate cancer (mCRPC) [130, 131]. Furthermore, another potent small molecule inhibitor, GSK2816126 (Table 1), which showed remarkable preclinical potential entered phase 1 clinical trial for the treatment of lymphoma and solid cancers but proved to be unsuitable target for inhibiting EZH2 due to its unfavorable pharmacokinetic profile [132]. More than 50 small molecule inhibitors for EZH2 are in preclinical development [133]. A few are highlighted in Table 1.

Epigenetic enzymeSmall molecule inhibitorsStage of developmentCancer treatmentReference
Lysine methyltransferases
EZH2EPZ6438 (Tazemostat)FDA approvedEpithelioid sarcoma[32]
Phase 1 clinical trialVarious lymphomas and
[ClinicalTrials.gov] Identifier: NCT03010982
GSK2816126Phase 1 clinical trialLymphoma and advanced solid tumors[132]
CP1-1205Phase 1b/2 clinical trialB-cell lymphoma, advanced solid tumors metastatic castration-resistant prostate cancer (mCRPC)[130, 131]
EI1Preclinical stageB-cell lymphoma with Y641 mutation[141]
EPZ011989Preclinical stageLymphoma[142]
UNC 1999Preclinical stageDiffused B-cell lymphoma with Y641N mutation and MLL rearranged leukemia[143, 144]
hDOT1LEPZ5676 (Pinometostat)Phase 1 clinical trialAcute myeloid leukemia (AML)/acute lymphoblastic leukemia (ALL)[137] [ClinicalTrials.gov] Identifier: NCT03724084
EPZ 004777Preclinical stageMLL rearranged leukemia[136]
SGC0946Preclinical applicationMLL rearranged leukemia[145]
Protein Arginine Methyltransferases
PRMT5GSK3326595(formerly EPZ015938)Phase 1 clinical trialSolid tumor and non-Hodgkin’s lymphoma[ClinicalTrials.gov] Identifier: NCT02783300
JNJ-64619178Phase 1 clinical trial[Clinical Trial identifier: NCT03573310]
EPZ015666Preclinical stageMantle cell Lymphoma[135]
PR5-LL-CM01Preclinical stageColon and Pancreatic cancer[114]
LLY-283PreclinicalOvarian, lung, breast, gastric, skin, and hematological cancers[140]

### Table 1.

List of representative small molecule inhibitors for EZH2, hDOT1L, and PRMT5.

### 4.2 Small molecule inhibitors of hDOT1L

There are over 20 hDOT1L small molecule inhibitors and can be categorized into four groups based on their mode of action: (i) SAH(S-adenosyl-L-homocysteine)-mimicking compounds; (ii) benzimidazole or (iii) urea group-containing compounds; and (iv) carbamate-containing compounds [134]. The activity of human homolog of yeast DOT1L or hDOT1L is mostly dysregulated in a subset of acute myeloid leukemia that has MLL gene translocation. This results in an onco-MLL protein which aberrantly recruits hDOT1L to the promoter of MLL target genes. Together with other transcription factors, hDOT1L drives the overexpression of HoxA9 and HoxA7 which leads to leukemia [135]. Hence, the need for inhibitors of hDOT1L. The first potent inhibitor of this HMT was EPZ004777 (Table 1), but it failed to progress through clinical development due to poor pharmacokinetics [136]. Another small molecule inhibitor, EPZ5676 (pinometostat) (Table 1) with Ki of less than 0.08nM, was shown to have improved pharmacokinetics and has now completed phase 1 clinical trial [137].

### 4.3 Small molecule inhibitors of PRMT5

PRMT5 is overexpressed in a several types of cancers. There are currently over 50 PRMT5 small molecule inhibitors, and PRMT5 is emerging as a hotspot for cancer targeted therapy [138]. Some small molecule inhibitors of PRMT5 are currently undergoing assessment in phase 1 clinical trial for non-Hodgkin lymphomas and solid cancers include GSK3326595 and JNJ-64619178 (Table 1). PRMT5 has also been implicated in the progression and metastasis of pancreatic and colon cancer. As a result, a lead compound, PR5-LL-CM01 (Table 1), has been discovered by our group to have more potent inhibitory properties compared to EPZ015666 in pancreatic and colon cancer cells [114]. However, EPZ015666 showed high potency in vitro and in mantle lymphoma cells with an IC50 of 22nM (Table 1) [139]. Another potent inhibitor of PRMT5, LLY-283 (IC50 = 20 nM) (Table 1) showed an outstanding inhibition of breast, lung, skin, ovarian, and hematological cancer cells’ proliferation [140].

## 5. Conclusion, perspective, and future directions

Taken together, in this chapter, we discussed the important roles that epigenetic enzymes play in a variety of cancers. We also summarized several popular methods currently used for screening small molecule inhibitors of epigenetic enzymes. As shown in Table 1, we provided a list of representative small molecule inhibitors of HMT that are either FDA approved, or at preclinical or different stages of clinical trials. Notably, compared to the well-developed HDAC small molecule inhibitors, the development of small molecule inhibitors for HMTs is a rising and cutting-edge drug development area. We can envision that in the next 5~10 years, intense attention will continuously be drawn to the discovery of HMTs small molecule inhibitors. We have no doubt that many HMTs small molecule inhibitors will be shifted into clinical trials, more will be approved by FDA, and most likely, more members of HMTs will be targeted for cancer treatment. Additionally, it is very possible that novel HTS methods will emerge, which will further accelerate the discovery of anti-HMTs drugs. Moreover, it is viable that the clinical indications of HMTs small molecule inhibitors could be further expanded to other diseases beyond cancer. In summary, the development of new classes of anti-HMTs drugs will offer brand new and exciting opportunities for diseases treatment.

## Acknowledgments

This publication is made possible, in part, with support from NIH-NIGMS Grant (#1R01GM120156-01A1) (to TL), and NIH-NCI Grant (#1R03CA223906-01) (to TL).

## Conflict of interest

The authors declare no potential conflicts of interest.

## How to cite and reference

### Cite this chapter Copy to clipboard

Aishat A. Motolani, Mengyao Sun, Matthew Martin, Steven Sun and Tao Lu (July 24th 2020). Discovery of Small Molecule Inhibitors for Histone Methyltransferases in Cancer [Online First], IntechOpen, DOI: 10.5772/intechopen.92830. Available from: